Technology stops for no one. Software is advancing at a phenomenal pace, and along with it, user expectations continue to grow and grow.
Building, supporting, and scaling modern software can be a daunting task – even for behemoth software organizations. In this article, we are going to review some recent software fails by digital giants and highlight key takeaways for your organization.
By the end of the article, you will understand that even industry-leading brands fail. What matters most is that your organization follows the best practices to minimize the impact of your failures.
Modern Security Breaches
Users are demanding. We touched on this topic in our post on personalization in software. The data shows resoundingly that consumers are willing to trade their personal information for a more personalized experience.
But there is a downside to all of this personalization. As more and more businesses collect user data, the consequences of a data breach or employee error grow.
Every few months there seems to be a news story on another data breach. Most recently, in July 2019, Capital One suffered a data breach that affected 106 million credit card customers. Upon closer inspection, it seems that a misconfigured Amazon Web Services (AWS) user permission was to blame.
Other recent breaches include: Quora (100 million users in December 2018), Marriott and Starwood Hotels (500 million visitors in November 2018), and MyFitnessPal (150 million users affected in March 2018).
We would also list Facebook, but there are quite frankly too many incidents to report. For example, the fact that data on 500+ million users were left exposed is not the largest blunder they have faced in the last few years.
Key Takeaways for You
Security needs to be a rising concern for every organization. Implementing regular security scans and software updates is a must. The degree of this will depend upon the size and scale of your digital presence.
Large organizations should:
- Perform regular security scans for new application vulnerabilities
- Keep all server software and frameworks up-to-date
- Implement security best-practices for coding and new organization software
- Audit legacy applications to determine existing security risks
Small organizations should:
- Have a security-first mindset when adding new software to their organization
- Be wary of free plugins and themes
- Hire a consultant to perform server software updates every 6 months
Modern Examples of Bad Practices
There are also a number of incidents where bad software development, deployment practices, or poor oversight, led to catastrophic errors. Here are a handful of examples.
In July 2019, CloudFlare suffered an outage that rendered a large number of websites inaccessible. According to the company, the outage was caused by a CPU spike due to “a bad software deployment”. Luckily, CloudFlare has great deployment practices and infrastructure, and they were able to mitigate the outage and get all of the sites back online within 27 minutes.
Earlier in 2019, Myspace made a serious error. (And yes, Myspace still exists.) The company shared that 50 million songs uploaded by 14 million artists between 2003 and 2015 were lost! Like forever. Myspace apparently does not believe in data backups.
If you are ever having a bad day, remember that you didn’t delete 12 years worth of your customers’ data on your platform.
A few years ago, Amazon Web Services (AWS) also had a problem with its Simple Storage Service (S3). The problem, reportedly caused by a typo, took down major web services including Trello, Quora, and IFTTT.
Here is a direct quote from AWS: “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended,” Amazon said. “The servers that were inadvertently removed supported two other S3 subsystems” (source).
Key Takeaways for You
Any organization scaling software should implement modern best practices for their deployment process. Approach every release with the mindset that everything will break, and make sure that you have backups and fallbacks in place. Again, the complexity of you devops should scale with the size of your organization.
Large organizations should:
Architect a proper CI/CD pipeline, leveraging testing automation where possible
Build and adhere to a release cycle
Automate backups
Have a plan outlined for hotfixes
Always have a fall-back plan to minimize downtime in case of a code-splosion
Small organizations should:
Leverage version control for all of your software, no matter how small the project
Perform regular backups
Setup a staging site for all websites / applications so that you can test updates before they are released
Conclusion
Some of the examples above are easily avoidable and stem from poor software development practices. Others are simply bad luck or good ole fashion human error.
Regardless, when you build software, you need to spend time building the right infrastructure to surround your product. Notice how CloudFlare and AWS were able to identify and solve issues in a handful of hours. Whereas MySpace lost 12 years of user data forever.
The idea is simple: When things break – and they will! – you need a strategy, a plan, a fallback. This plan starts before the code is even written. You need a solid testing and deployment process, with monitoring and logging so you can understand the health of your system.
Furthermore, when deploying new software, make sure that you do not rush. Your goal should be maximum stability and the best possible customer experience. As we have seen, even small bugs in code or entering the wrong server name can have drastic consequences.
Lastly, we do not share these examples to scare you. If things do break, don’t panic. It even happens to the most successful companies in our industry. Even the gods fail sometimes.
Looking for help with your tech infrastructure? Contact Tragic Media today and get a free consultation! We don’t want your next software project to turn into a tragedy.
