According to a senior staff engineer, Matt Singer of Twitter Inc (NYSE:TWTR), it is difficult to keep track of all the data on the social media platform. Every Tweet comprises more than 100 data points. The data generated from every retweet, link click, unfollow, and other actions help the deep learning systems and analytics that serve advertising, operational, and other critical functions. Such events are non-stopping and exceed 1.5 trillion events daily.
Difficult to handle hyper-scale demands
Twitter depends on Big Data Analytics Software on Hadoop clusters to get business insights that help to grow/ manage the vast global network. To maintain its lead in the streaming information service, the company sought the help of Intel Corporation (NASDAQ:INTC) to minimize the rising storage costs and improve performance.
Intel achieves a breakthrough in Hadoop run times
The collaboration between Intel and Twitter paid dividends. Intel engineers suggested using selectively caching and storing the files temporarily on SSDs and enhance the processor density by six times. This innovative approach helped Twitter to achieve 50% faster run times on Hadoop, and reduce the hard disks by 75%. It also helped to reduce TCO costs by 30% and brought new reference architecture for the server clusters for stable growth.
Slow Analytics with slower drives
The size of the hard disks has increased, but not their I/O speeds. Hard disks perform better when tasks are performed one at a time. They fail miserably when handling multiple tasks at the same time. The same is the case with Hadoop workloads on regular HDDs running at 7200 rpm. Slower I/O speeds limits hardware and architectural choice of an organization. Despite adding storage and expensive servers to Hadoop clusters, Twitter hit the roadblock.
Intel engineers during a discussion said using intelligent caching in Hadoop clusters with the help of Intel CAS to solve the issue. The technical solution is a miracle and helped Twitter to reduce storage and maximize performance. Instead of caching the entire data, the solution uses smaller amounts of critical path working data. Intelligent caching paved the way for Twitter to solve the issues.
Twitter fixes the outage issue
Twitter has received 4,000 reports of outage from nations, including India, Japan, and Canada, in the early morning on Wednesday. The company said that it has fixed the issue, and users can access the platform.
