Hosted Search API that delivers instant and relevant results from the first keystroke
March 3rd 2016, 7:00 pm
How does Algolia do indexing? Does it index all potential keystroke combinations?
To explain, here's an example of how I built a super fast autocomplete service once: If the word that had to be indexed was
paper, I would index
r as well as
er as well as
per as well as
aper and finally the full word,
Now if I type in
p, it will mach with
p is indexed,
pa will continue to match
par will not match
paper since it's not indexed.
Does Algolia do something similar?
What is your strategy to manage the huge number of indices being created per word assuming you follow a similar strategy?
Also, do you mind sharing what your storage strategy and engine is? My assumption is that you have machines / instances with huge amounts of RAM in order to make lookups super fast and using the above strategy of indexing it can even work on something like Redis. Then the only bottleneck would be the network, so using something similar to what CDNs are doing, you are placing machines close to where the searches are being done in order to get the type of speed you are getting?
We don't compute all keystroke combinations. Only prefixes are important in an instant search (you are typing the query, so the last word is always partial). Ideally, we would like to compute all prefixes, but this is too expensive and would greatly increase the size of the index. For example, if you type "a", it should match all the words starting with "a" (potentially thousands). In order to solve that, we try to always query a maximum of 10 words, and we introduce "virtual words" to our dictionaries that contain some inverted list for prefixes. These are efficiently computed on-the-fly at indexing and only require a few KBs of memory. We plan to explain this data structure in more detail in a blog post. It is not easy to explain it in an AMA. :) For performance reasons, we try to always have an index in memory. All our plans are designed for performance with an index fully stored in memory and duplicated on at least 3 different machines. This is also a very good way to ensure indexing will not impact search performance.
Does Algolia take natural language into account? If I search for "book" (as an example), does it know whether I'm looking for the noun (I'm reading a book) or the verb (In order to attend, you need to book)?
We do not apply part of speech tagging to documents we index, but we do rely on the record structure (the different attributes, nested-objects in the JSON). For your book example, you will always have a category or structure that returns actual books before returning a verb in a description or in the full-text. Part of speech tagging is CPU expensive, it’s not 100% accurate and it’s not as good as structure in records :) That said, we rely on lemmatization dictionaries to ignore singulars/plurals and let you integrate synonym dictionaries if you have such a resource.
Julien Lemoine : What motivated you to start Algolia? How do you think Algolia is going to disrupt the way developers implement search functionality in their apps?
The very early beginning was in early 2012. I was doing some freelance to help a customer implement a very good instant-autocomplete. After working for more than 10 years on search engines, I had the impression that no real progress had been made: very good tech on the market but all very complex to use and tune to actually get great results. I discussed some different ideas with Nicolas (cofounder), and we decided to use our expertise to solve search problems in a different way. Algolia disrupts the way engineers build search by solving all common pains while having a 100% white box approach. Nothing is hidden, the developer maintains complete control and the search is easily customizable. You can expect a lot of new things coming on this side :)
As a complement to what Julien just said, I’d add that we had the same idea of the kind of companies we wanted to build. We actually sat down for half a day and discussed the culture we wanted to build before giving it a go. Knowing that we share the same values was very important for us :)
I can see that you guys are using Angular in your Dashboard. Do you have any plans to upgrade to Angular2 or React? I am asking because vvo is involved in some good React projects like
As a matter of fact, we are in the process of incrementally upgrading the Algolia Dashboard to React + Redux – you can already see it live in the Api keys page! Great development experience so far (hot reloading, React & Redux dev tools, etc). We’re also using React on instantsearch.js.
We decided to go with React because we love the ecosystem around it and it’s really powerful in combination with Redux and Webpack. That’s also a really good way for the whole team to share knowledge and have a common playground. We’re intrigued by Angular2 as well, but it’s not currently a viable choice maturity-wise.
Nice to see you here. My question is : How does Algolia manage to optimize everything at milliseconds level?
Short answer: because we are obsessed with it every minute of every day! Long answer: by analyzing all the layers/components with milliseconds in mind--the network, the hardware stack and obviously the software stack. No compromise! Everything is designed to be highly efficient, and our goal is to make it impossible to do better! For example, we embed our query code directly in nginx using a C++ module. In 99% of use-cases, the index is directly memory-mapped and the query is executed on the serialized data-structure on the fly; the result is then directly returned by nginx.
We try to optimize at the milliseconds level in every part of our code. The engine and the network being the most important parts. But we also optimize our frontend tools like instantsearch.js.
In an interface where you render results on screen at every keystroke, you need to carefully optimize your code to only render what needs to be rendered.
What's your secret sauce? :D
More seriously, here is how we tackled the ranking issues every website/app experience while trying to build a relevant search: https://blog.algolia.com/search-ranking-algorithm-unveiled/ :)
What are some unique things about your company culture?
I'm not sure what is unique. Every company has a unique culture. Every new hire changes the company culture with what he/she brings.
Still, one important part for me is ownership. I know that most startups say that already, but I've never worked in a company where I had so much freedom into pushing stuff into production quickly and easily. Ownership grants you freedom, but it also comes with accountability, which is fine, because failures are the only real way to learn.
To illustrate that, I feel like the work I actually do is what I would do on my freetime for fun anyway.
Our culture is made up of our values. The values we share between us are trust, care and excellence. Our values could just be seen as any other written words, but we truly believe in them and it translates into our culture. We live by these written words everyday and incorporate them into our day-to-day operations. Beyond the organizational values, Algolia is really an environment in which you can be yourself. We learn from one another, and it’s all about teamwork and collaboration. We have so many different personalities on the team, and everyone is encouraged to push each other to be better. Our organization is pretty flat, and we let our people take ownership on any projects they want. We are fully transparent with the company because we trust everyone; we share our compensation with the team, and our founders share their monthly update newsletter to the investors with the team.
We have company wide happy hours every Thursday and a live stream that’s always on between the two offices. We encourage travel between the Paris and SF offices so we can see each other in person as much as possible.
Long story short, it might sound cheesy, but it all makes for an open, fun work environment that is unlike anywhere I’ve worked before.
What a good question Mayank. This is not something that is easily answered because it feels like Algolia is a new company every 3-6 months. That being said we do have strong core characteristics like Trust, Honesty, etc... but what I think makes us unique is our ability to openly give and accept feedback - every day feels like we're moving 100mph, which means that everything we do we want it to be perfect. So that being said, we want everyone to be able to be 'challenged' on their proposition of a project or idea and make sure that we look at it from all angles. It has allowed us to really refine and improve the actions that we take on everyday tasks.
In terms of pricing, can there be a "Start up" plan which will be between Starter and Growth?
I am a huge fan of Algolia search. Was wondering how big is the Algolia team?
How do you handle 300+ servers in 35 datacenters (in 15 regions)?
Via automation, monitoring and alerting :) We automate everything we can (we mainly use Chef), we monitor like crazy (currently we monitor over 250,000 metrics, and more than 10k points/seconds and still growing) and we have alerts on everything that could be wrong, even slow indexing or slow queries. Despite all of that, we add new metrics every week. The logistics are a pretty important part of our jobs. Finding and dealing with a big number of providers worldwide is part of our job.
Automation, automation, automation (and some automated testing). First of all, all our service are resilient. We try to have 2n+1 clusters for every service we provide. Because Algolia is not only a search engine, but also an analytics platform, a monitoring system, a website, and so on.
Hey Team Algolia, good to see you here. My question is how did Algolia convince its very first client to use Algolia in their search? What was the sales pitch like?
Our early customers were the ones we converted from the beta. So the question becomes—how did we get our first beta customers? We reached out first to our network and hustled from there. We did a lot of demos and got tons of ‘nos’ or worse, ‘maybes’. But then we convinced a first one, and then a second, and the product improved, the referrals started to come in and it got much easier.
Pitch wise, we focused mainly on the speed. We spoke a lot more about the typo-tolerance than we do today, and one thing that really hasn’t changed at all: we demoed a lot!!! Actually, today, we still (nearly) never use slides in our sales pitches (well, we maybe should sometimes!)
Considering that we're moving towards an internet that is getting more and more fragmented via apps (genius.com for lyrics, product hunt for new products), in future these apps might become the go-to place for searching something. Now, Algolia is powering the search functionality in these apps without taking any personal data and working in a device-agnostic mode.
So my question is this - aren't you one of the major threat to Google?
We’re both doing search but in a totally different way. Google indexes web pages and try to detect which pages are relevant based on 200+ ranking web criteria.
On the other hand, we’re helping developers to build their internal search engine by providing them all they need to customize the ranking/relevance of their search based on business knowledge/data. Most of the time, the data you use to customize the Algolia ranking is not public and cannot be guessed by Google: number of sales, number of page views, etc...
That would explain the cute little car (http://www.google.com/selfdrivingcar/images/home-where.jpg) that’s always parked outside of our office. ;)
More seriously, we see Algolia as complementary to Google. Sometimes you want to search for general information across multiple sites, and Google really is still the best at that. And sometimes you want to search just through a specific site, and that’s where we think we’re creating a much better experience--both for website/app owners and for their users.
I hope you don't mind me asking multiple questions... What can you tell us about the internal data storage? What DB or architecture do you use?
How hard was it to build initial Algolia team? What advice founders can give to first time entrepreneurs regarding hiring?
Hiring is a very difficult topic. We have a very long hiring process. I had more than 8 interviews, a full day with my potential colleagues and a home assignment. I think the best advice is : “do not underestimate hiring”. If someone has a reserve on any future employee, just stop the process and seek the next one. One bad hiring can costs you a lot.
It’s true that our hiring process is quite long. But this is because we really want to make sure that we, as a team, would like to work with the new hire.
If we ask our potential hires to spend a full day at the office, it’s to show them how life at Algolia really is, so they can see if that it is what they had in mind. On the other hand, it lets us spend some time with them, discussing various topics and have a “gut feeling” if we’d like to work together.
There’s something unique in working with smart and brilliant people who share your same passion for the product and love to get shit done while having fun at the same time.
It’s definitely important for us to know the person behind the candidate and see how they can complement your daily job with their skills and personality. That’s why it’s so hard to hire the right people and why it takes so much time to build the right team.
How do you collaborate between your Paris and SF offices? Is it hard?
Or Slack, or Google Drive, or a lot of others SaaS tools we're using :). Mostly, we try to make sure all important information is written somewhere and not only discussed physically on one side of the ocean.
We also do weekly 30mn meetings (via video) with the whole team to keep everyone in sync of what is happening in the company.
It has its challenges, but it also has its benefits, too. I’ve spent equal time in each office and you can definitely get a feel for small differences between the two.
The SF office was only started within the past year, so it’s much smaller than the Paris office. (We’re growing all the time so I don’t know the exact numbers, but it’s something like 35 in Paris and 10 in SF.) All of our devs are in Paris, while many of our business and marketing people, including our CEO and CFO, are based in SF.
We use Slack and video conferencing liberally. We also use other tools for collaboration like Asana, Google Docs, Github, etc. One of our core values is transparency, so making sure we share information across all offices is really important.
One of the big challenges is what we do in SF after Paris has signed off for the night. Since we currently have all of our devs in Paris, we request at least one dev cycle through SF at all times. That way we can answer support questions quickly at all hours.
We have corporate apartments in SF and Paris a walking distance from the offices, so there is a lot of travel between the two. This gives the team a great opportunity to get to know each other in person and share the cultures between the two.
That’s the challenges, but the benefits are great, too: we can draw on the cultures and the talents of France and Europe as well as the United States. It makes for a very refreshing company culture.
It is hard because you can easily feel the disconnection between the two offices.
What we ask our people is to maximize the opportunities for collaboration between the two offices. Right now the SF office is a lot about HR, culture, marketing while the Paris office is about engineering.
Since we have a blog: https://blog.algolia.com/ we leverage this as a opportunity to collaborate between engineers and marketing team.
A very important part to me is traveling. We ask every employee to travel at least to the other office, SF or Paris. This is working very well because as soon as you travel to the other office, you can feel and experiment a direct link with people.
To ease this process, we have corporate apartments in both cities, near the office.
One thing that has not been mentioned yet: the culture. We really value ownership a lot and that’s enabling us to make decisions without have to refer to each other all the time. We trust people to make the best decisions and to make sure they can do so, we are very transparent about basically everything that’s happening and prefer to over-communicate than keep things to ourselves. We also encourage everyone at the company to travel to the other office at least once a year. We have usually between 2 and 5 people from Paris in SF at any time, for example :) That’s helping a lot and makes sure everyone can get to know each other! Oh and we only speak English, including in the French office where we have currently 5 non-native French speaking employees \o/
How is Code Review done within Algolia team? Do you guys use any tools like Gerrit?
There are always a lot of different projects going on internally at Algolia, from the core engine to the public API clients. All projects follow mostly the same flexible process: we use feature branches off our master branch in git that are then merged back into master after a Pull Request. This Pull Request allows us to do some code review asynchronously.
All commits are public, so anyone can review them. We usually wait for at least one collaborator to comment with a “LGTM” (Looks Good To Me) before merging. If some code is unclear or could be improved, we leave comments (directly in GitHub) until the PR is ok. This allows each of us a chance to have a look at the integrated code and learn/ask questions before it gets merged.
To keep the human verifications to a minimum, we heavily rely on automated tools (linters, unit tests) and run them automatically through TravisCI on each PR. If the automated build fails, we usually do not even bother to check the code and just wait for the commiter to fix it. Only once the automated tests pass do we start checking the actual code.
We also heavily use Slack internally to discuss all possible matters. We have GitHub integrations into specific channels that let us know when new commit/PR/comments are added. We’re not doing much pair programming today in terms of code review, and this is something we’d like to do more in the future. Because we have a team distributed between Paris and SF, it is always better to be able to do things asynchronously.
Do you go to mars? In your Fusèe?
It's a guy who just went out of prison. He seeks a job, and every interview he goes to ask him if he knows how to speak english. But he can't. So he searches for english courses, but he is broke. One day he found a very cheap course on the newspaper. So he goes to the place. Rings, and a guy answers him:
- He asks: "is it here for the english courses?"
- The guy answer: "If, if, between!"
How was your experience with YC? What was the key takeaway?
Short version: awesome! It would take a long time to cover all the great things about YC like the really smart and experienced partners, incredible batch mates you’re going to have, and the very helpful network of alumni… If I had to only pick one thing, I’d say that the experience of moving the full team (6 of us at that time) to the Bay Area was the most impactful. We worked and lived together for 3 months out of a house we rented in Menlo Park. Participating in a YC batch is very intense, and doing that all together from the same place, basically working night and days for 3 months, contributed a lot to our team and culture.
We were 6 people sleeping/working/eating/living in the same house during 4 months. It turns our than as soon as you work 18 hours per day, only focusing on your product & users; ensuring you keep a 10% growth from one week to another; you really understand what are the next steps you need to build to make your product better.
Why should I use Algolia if I can use full-text search, ElasticSearch or Solr for my app?
We redesigned from scratch how search engines work, mainly to fix the following issues:
Relevance: we provide a different approach to relevance that gives you a predictable ability to mix textual relevance and business date (We know well the pain of trying to tune the big relevance mathematical formula: you improve one query but you degrade a lot of others. We use a n-dimensions sort, also known as tie-breaking, to make sure you always have control over your relevance.
Performance: we optimize everything at the millisecond level, and our goal is to make it impossible to do it faster :) We have a unique infrastructure deployed in 36 data centers across 15 regions and software optimized at the millisecond. Focus: we’ve designed our engine to reproduce the experience you would get on Google or Amazon on any website, focusing on the user-facing search experience. The one you use every day, handling typo-tolerance & providing results in real-time -> it turns out that this is a lot different than analytics, dashboarding or alerting that the competition is focusing on.
UI/UX: we don't only provide the engine but expert advice and libraries to deliver a great UX
Dev-friendliness: we are developers, so we know the joy of using a well designed API
Support: the developers building the product are the ones doing the support, and we don’t plan to change that anytime soon
If you're building a user-facing search bar and you want a search experience that looks like you have on Google and Amazon, we're definitely the best fit. That's because we've designed our engine to handle as-you-type search & typing mistakes and injecting business data in the ranking algorithm to provide your users the best text & business relevance.
We're also Elasticsearch users, we use it for its analytic/aggregation capabilities: we're building analytics on top of our end-user queries (just to be clear: we're NOT using Elasticsearch for the Algolia search API). It's a very good piece of software but we don't use their ranking or full-text search capabilities.
Instantsearch.js is great. Are you guys working on any other open source projects?
Thanks! instantsearch.js is going well with more than 150 clients using it. We are getting a lot of feedback and will soon make it more compatible with React for example.
All our work is open source BUT the actual algolia engine, mostly because it does not make sense right now to open source it.
It's really great to work on everything open source on a daily basis.
Actually we released today our community website, listing most of our most important open source work: https://community.algolia.com/
Our next open source project should ve https://community.algolia.com/docsearch/, we already open sourced the js part: https://github.com/algolia/docsearch.js and we should be releasing the scraping part soon :)
Hey guys! great job on algolia. We are a nonprofit edu-tech company trying to connect and educate young kids using games.
My question: Is there a way Algolia can be used to set up a "visual search” inside our apps where when someone taps on a character in the game, it would search and give images related to that character(kinda based on related tags).. and how difficult would it be to set up something like this into our existing apps?
Yes building a game based on a search engine is doable. We have done one experiment with Imagga image tagging few months ago: https://blog.algolia.com/robots-vs-humans-who-will-emerge-supreme-in-the-battle-of-the-image-tags/
What's the best way to migrate the search implementation from a custom MongoDB solution to Algolia? Is there a way to export data easily from Mongo or some other DB to Algolia?
We propose a mongodb connector that will import and synchronize all your data on Algolia. To do that we’ve built a MongoDB connector acting as a MongoDB replicate forwarding every single operation to your Algolia index. That being said, from our experience doing a simple MongoDB collection ⇔ Algolia index mapping is not enough and we highly recommend using one of our API clients to synchronize your MongoDB collection with our engine using a few lines of code.
How do you test your React components? And how is Redux working out for you?
Testing React components is tough, you've so many way to test them :) We've written a blog post about it here: https://blog.algolia.com/how-we-unit-test-react-components-using-expect-jsx/
Redux is working great so far. Separating data logic from the views/component is liberating. It is also really easy to test the actions and reducers because they are just pure functions. We are still trying to find ways to decrease the boilerplate though (we have something in the works for this :).
Re testing components: on the website, we use a combination of airbnb’s enzyme and our very own vvo’s expect-jsx
We started testing React components with expect-jsx: https://github.com/algolia/expect-jsx that we built. But then testing for events is a bit tricky, we explain it on our blog https://blog.algolia.com/how-we-unit-test-react-components-using-expect-jsx/.
Right now the airbnb project called enzyme http://airbnb.io/enzyme/ seems to be the most promising testing library.
I also believe the React team is working hard to make all this easier. Right now it's a bit tricky to know how to test React.
We are currently unit testing our reducers by using mocha, chai.js and sinon.js while we're using expect-jsx for integration / functional tests.
So far Redux has proven to be very helpful in helping us handle state updates without undesired side effects: we managed to remove most of the local state from our UI components. The only downside we found is a little bit too much boilerplate code, especially when handling asynchronous flows. It’s definitely working well for us, we 💖React and Redux!
How is your recruitment process? Do you help to get a visa? Asking, because Facebook just recently demystified its process (and of cause for personal reasons too).
We are growing super fast and, as a result, we invest a lot in recruiting. It depends a lot on the position, of course, but our recruitment is a mix of outbound contact, referrals from friends/colleagues, and people who are excited about Algolia and want to work with us.
Most of our team right now is in Paris, with the rest in SF. By the end of the year, we’re hoping for half and half in each city.
The recruitment process was among the best that I’ve experienced before. The team was very thorough (I spoke with 6 people before flying to Paris for a day on-site) and made sure that we were good fits for each other, while at the same time eschewing the “trivia” that you see in a lot of tech interviews. Throughout the whole process, I really felt like the team was being supportive and almost cheering for me to do well, even as it was clear that they had a high bar to clear. It was a good insight into how the team works on the day to day.
I can say confidently that for the right people, Algolia will help with the visa process because I’m going through it right now. ;)
As a San Francisco-based employee, I’ll just add that smaller can be better! Team size! I mean team size!
Also, a key part of the interview process is a presentation to the whole team. The subject varies based on your role and it can be a bit intimidating, but we’re a supportive and curious group. I think it works well for our candidates and for our selection process.
Considering the scale at which Algolia operates, how do you handle DevOps? How do you make sure you are deploying big changes without breaking anything?
This is a pretty big question and a daily concern of ours. We try to always think about the consequence of a new code and deploy it gradually to avoid unforeseen impact. That said, a mistake can happen. You can count on us to be as transparent, reactive to fix the issue and help you as if we were in the seat next to you.
Testing, testing, testing. We automate a lot of stuff, including unit & integration testing. Furthermore, if we need to do a breaking change, we deploy it incrementally and check our metrics to ensure everything is working well. For example, if we need to change the contract between 2 services. We deploy the new version of the producer, adapt the consumer to this new contract, then remove the old code from the producer.
Most of our software is tested automatically for regressions. It does not means nothing will ever break but we try our best to make our code rock solid.
For example all our code on github is tested automatically using travis. For some projects we also use saucelabs to start real browsers and do unit or functional tests with them
One very important part is the API. We strive to always make it backward compatible. If we introduce a new feature or new syntax, we make sure the previous syntax is still working.
We’d rather update the documentation to stop mentioning an old feature than removing it from the engine. Sometimes, when we really need to change a behavior, we inspect our logs to get the list of potential affected customers and proactively contact them to warn them about the coming change.
Talking about our website and dashboard we deploy from 5 to 20 times per day using capistrano and Travis CI without any particular problem. It’s a very simple setup but it works pretty well in practice: if something goes wrong we just rollback to the previous working version. We rely a lot on automated tests but when things goes wrong we usually fix the issue as soon as possible.
What are some major websites that uses Algolia? How many sites, in total number, use Algolia?
We have more than 3000 active applications using us. We process more than 30 billion API calls per month. Algolia customers include: Vevo, Periscope, Quiksilver, Medium, Product Hunt, A Little Market (an Etsy company), CrunchBase, HackerNews, Genius.com and Livestream. We unfortunately cannot comment on our biggest customers because of NDAs... this is so sad :(
We actually have an NDA with a lot of our big customers, but the main public ones are Medium, Periscope, Vevo, Product Hunt, CrunchBase, DigitalOcean.
Regarding numbers, I think we are live on around ~1300 paying customers (and one customer can have several websites/mobile applications). We also have more than 4500 (free) Hacker plans.
Is it true that Algolia was first made for Mobile Apps? If that’s the case what made you change your mind?
Yes that's true :) We first designed our engine to run on mobile phones: building a small SDK you would embed in your mobile application. After a few months, we had a very good search engine, running on the cheapest Android phone (with no RAM, no CPU, no Disk) but the business was tough. It turns out that our users had exactly the same needs on their websites; so we just pivoted to SAAS: kept the same engine, hosted on clusters of 3 high-end machines \o/
Yes it is true! We basically saw so many unanswered questions on StackOverFlow about how to implement Lucene in mobile apps that we thought it would be a great problem to tackle :) The thing is the market was not as great as we expected/hoped or maybe more difficult to penetrate. We did, however, get a lot of positive feedback about the UX of the search with people wanted to do the same thing online. So we pivoted :) And that was one of our best decisions! We rapidly realized that all the choices we made for mobile were great fits for user-facing search! Because of the hardware constraints of mobile, we couldn’t use the traditional search approaches used by Elasticsearch for example. The form factor also drove a lot of our decisions (instant display of results as you type, typing mistake tolerance, factoring geo-distance in the ranking, etc.). In the end it shaped the technology and is a big reason for our current success :) We probably never would have made the same choices had we started on servers.
How do you manage performance budgets - if you have any? I see in ChromeDevTools some interesting numbers, but I wonder how do you care for keeping response times low and payload short?
P.S. You need Node.js developers in SF ;-) hint hint
In terms of engineering what do you look for in software developers who want to join you?
Culture fit first, we like people that want to think about the most important thing for the company (we are 100% transparent on all topics). Then expertise, as a team we search for people that are better than us on at least one topic and that bring something new. The hiring process includes one day in the team with a final presentation on a topic you are passionate about.
We’re looking for talented, passion-driven developers which love to experiment and deliver features.
We have lots of room for personal projects as well, so ownership is definitely one aspect we really appreciate. Humble and grit are also very important personality traits we love in a person: this keeps us motivated and on the same page.
What was the most challenging thing for Algolia while scaling up?
We're definitely still scaling up, and have had to overcome a lot of different challenges along the way. At about ~45 people, we're starting to need more internal processes to make things work smoothly. That said, we're always trying to put Algolia in more people's hands — employees, customers, and end-users.
Hey! I understood that Algolia architecture is not "elastic" at this stage. Is it a product decision? Do you plan to change that?
EDIT: sorry I should have been more specific about elasticity. I mean you do not allocate resources (scaling up and down) depending on real time usage at this stage, if I understood well Sylvain's great presentation during the PerfUG. Is it a business choice or do you plan to modify this in the future?
Elastic is a trendy keyword that can means a lot of things. We are elastic in several ways:
- we can add more indexing capacity (more clusters)
- we can add more search capacity (more machines with copy of data, our biggest user perform around 150M queries per day)
- we can distribute your search worldwide while ensuring a fast search everywhere (adding DSN replicates in a different region)
- we have all data in memory in 99% of cases but we have few use users where over-booking make sense to reduce costs (over-booking of memory)
At the end, elasticity depends a lot of the use case
We host ou starter to pro plans on high-end clusters, we usually host up to 100 customers on a cluster. When we install a new cluster, it is empty. Of course users can grow above the limit of the cluster, in this case we migrate some users on the new cluster, this is an operation which is transparent for our users and let them upgrade or downgrade. We ensure we always have more resource than the current usage (we keep around 25% of free resource on our global infra for scaling)
For our big enterprise customer, we have a dedicated infrastructure for them and they have the control on upgrade/downgrade. We also have a migration procedure to move from a pro plan to an dedicated infrastructure, you don't need to reimport your data.
With the growth we have, we add new machines in production close to every day. So we scale up the infra very often. There is no technical limitation to scale down (same principle than scaling up), we hope it will never be needed :)
Julien's comment answers it well :+1: We definitely plan to keep improving our abilities to scale up & down as fast (dynamically?automatically?) as possible -> it's always a matter of focus & priorities. Right now, our current elasticity capabilities covers 99% of the use-cases we're targeting.
Algolia is a hosted full-text, numerical, and faceted search engine capable of delivering realtime results from the first keystroke. Built with developers in mind, Algolia's powerful API lets you quickly and seamlessly implement realtime search within your websites and mobile applications.
Algolia examines all words within full-text fields in order to find the most relevant records. This capability allows Algolia to quickly search and return records from large volumes of data.
Algolia responds to search queries in milliseconds which enables a unique search as you type experience. This allows your users to think less about how to search and more about what they find. Exploring neighboring queries becomes easier, making content browsing more intuitive for your audience.
Shoot us any questions you want us to answer!