The Co-Founder Of Reflect Shares Technical Insights Into His Novel Web-Based Testing Product
Fitz is a co-founder of Reflect Software LLC. Reflect is an automated web testing solution that anyone can use. This novel solution enables us to test our web applications in the same way that an end-user uses the application, without the need to write any sort of code.
In a recent conversation, I had with him, he shared detailed technical insights into how he developed this much-needed product from the ground up.
So, without further ado. Let’s get on with it.
Tell Us A Bit About Yourself?
My name is Fitz Nowlan and I am a software engineer by training. I have a PhD in Computer Science with a focus on Networking, and I’ve worked at both “big tech” and start-ups in my career. I recently left my position as a Director of Engineering at a digital marketing start-up to co-found a new company named Reflect.
Tell Us About Your Product? What Pain Points Are You Trying To Solve?
Reflect enables developers and non-developers to create and execute automated regression tests on web applications. To create a test in Reflect, a user simply enters a URL and interacts with the website or application.
Reflect records your actions and converts them into an automated test that can be executed on a schedule or on-demand. If the actions cannot be executed successfully, Reflect notifies the user via email with information for diagnosing the failure, including a video reproducing the failure, and the network and JS console logs.
In spirit, Reflect fits the mould of a “point-and-click” test recorder, but in reality, it takes a much different approach. Reflect is entirely cloud-based. This means that the user does not need to install an app or browser extension on their computer; nor do they need to clear their cookies or modify their browser in any way before recording a test.
Users interact directly with Reflect’s cloud-based browser as though they were interacting with the live site. This approach avoids an entire class of test failures stemming from differences in system configuration, which often plague extension-based record-and-playback tools.
Lastly, the underlying philosophy of Reflect is to test the web application in the same way that an end-user uses the application. This means that Reflect is zero-code for its users, and relies instead on producing the highest fidelity test definitions possible by instrumenting the cloud-based browser.
The trade-off for losing programmability is the democratization of test creation and management in an organization: anyone who can use the application can now test the application.
Well, That’s A Much Needed & An Interesting Solution!! So, What Really Inspired You To Write This Product? How Did You Come Up With The Idea?
Our inspiration for Reflect was years of “pain” performing repeated manual regression tests at our previous jobs. Whenever we would deploy a new version of our web application, we’d spend significant time manually sanity-checking or “smoke” testing the application, which slowed our development velocity.
Eventually, we paid a lot of money to have a third-party provider test our application and give us the green-light on deployments. We realized it should be possible to record these interactive web sessions and replay them on-demand. The key insight for us was that we did not want to author these test definitions by hand, in code. That simply increases our codebase and doesn’t fully replicate the end-user experience of interacting with components and observing the visual appearance of elements.
Furthermore, code-based automated solutions, such as Selenium, require a maintainer with at least some specialized skills. We wanted anyone to be able to create and manage UI tests.
Would You Like To Talk About Your Customer Acquisition Strategy & The User Engagement & Retention Rate?
Our primary focus in the early going has been on direct customer outreach. We’ve had encouraging response rates from our immediate and extended personal networks, and have gotten to know all of our initial customers personally.
In addition to direct outreach, we’ve also focused on publicizing Reflect in online communities such as Twitter and Indie Hackers, as well as real-world meet-ups.
Our subscriptions are month-to-month, cancel anytime so that we learn immediately when we’re no longer providing value to our customers, or our application is falling short of expectations. Fortunately, we have retained every customer who has registered for a monthly subscription with us so far. Our customer service certainly contributes to this positive result.
What Is Your Monthly Revenue?
Less than $10K.
Okay, Let’s Talk About The Technology Stack You’ve Leveraged To Build Your Product
Both our web application and marketing site run in AWS because that is what we were most familiar with. Additionally, our payments processor, Stripe, offers AWS credits for new customers which enabled us to get up and running without any initial costs.
We use several AWS services: EC2, S3, CloudFront, Cognito, API Gateway, ECS, RDS, Elasticache and Secrets Manager. We’ve found that the AWS SDK is fairly consistent in its interfaces across services and it’s been very easy to incorporate a new service into our application (e.g., just a few hours for Secrets Manager).
The web app front-end is written with the Vue framework, and the back-end API/WebSocket layer is all written in Scala atop Http4s. We do nothing fancy on the infrastructure management side; we use a collection of bash scripts to manage our build artifacts and deployments.
What’s Your System Design Like?
Reflect is a desktop web application. The following diagram shows the architecture:
The Reflect web app is written in HTML/CSS/Typescript and communicates with our front-end web server over HTTP and WebSockets. Additionally, it establishes a VNC connection over HTTP directly with a browser session container whenever a test is being recorded or executed.
The web app server uses our internal management API service to perform any tasks related to browser sessions. This ensures that the Management service is the single source of truth for controlling browsers. Both the web app and Management service talk to our internal database and blob storage (S3), where necessary. The Management service also runs a few internal daemon processes used for various things such as triggering email notifications on test failures.
Browser session containers communicate with the end user’s browser through the aforementioned VNC connection, and they also communicate with the Management service for retrieving test data and saving test results.
This design allows us to deploy the three components independently of each other, which is faster than a single deployable, and obviously allows us to scale the units based on their specific load.
Would You Like To Elaborate A Bit On Your AWS Deployment?
We use AWS for everything because it’s reliable, cheap and familiar. When we were just starting out, we rented two small EC2 instances to separate our two services: the web application front-end, and the browser session executors. (The marketing site is a static site hosted on S3.)
The web application launches browser session executors on the second instance. As we’ve grown and we’re executing more tests, we moved to ECS clusters that can grow and shrink the size of their backing EC2 pools based on the number of browser sessions we need to execute. Total costs are less than $1000 per month due to dynamic scaling (i.e., we only need to pay for instances when tests are running or being recorded). EC2, ELB and RDS instances are our largest cost drivers.
What Are Your Scaling Strategies?
Since our system is a microservices architecture, the services can generally scale independently of each other with the obvious caveat that services that are co-located on the same backing EC2 instances share physical resources.
The heaviest lifting for the web app front-end is serving users who are managing tests and viewing test results. If we have a ton of users at once, we can add more web app containers. The management service is our internal API that manages the browser sessions containers. If we’re running a lot of browser sessions (either test recordings or test executions) then we can scale this service up by adding more containers.
The browser session containers run on their own ECS cluster and the only scaling here is how many backing EC2 instances we need to handle the volume of browser containers we have running at any given moment. We scale the EC2 cluster up when we approach our physical resource limits with the running browser session containers.
In all cases, AWS makes scaling the EC2 clusters easy with auto-scaling rules based on CPU and/or Memory usage.
Speaking of the product design process & I have a few questions regarding this:
If You Were To Rebuild Your Service From The Bare Bones, Is There Anything You Would Do Differently?
We have not yet discovered any major shortcomings with our original design. Our largest refactor to date was combining our test recording logic with our test execution logic to make it easy to launch a browser session that toggles between recording and playback modes.
What Are Some Of The Core Engineering Problems That You Faced While Developing The Product?
Two issues that come to mind are:
Networking configuration – we spent a lot of time getting the networking flows correct so that the end-user can interact directly with their live browser session container through a dynamically-established VNC session. This requires us to use dynamic routing rules through the load balancer and an always-on server management process to create and tear-down routing state.
In terms of learnings, it’s nothing profound, but the key to getting networking working is making the smallest possible incremental change and then testing connectivity repeatedly. We spent a lot of time backtracking because we would inadvertently make multiple changes at once to the flows or routing rules and one of them would fail. When all else fails, check your security groups (both for your instances and the services themselves).
Managing containers – we were quite familiar with ECS and containers from our previous jobs but our exposure had always been as either a typical microservice fronted by a load balancer OR a typical container daemon that repeatedly performs the same task without the state. In both cases, the container is stateless, which simplifies the handling of container lifecycle events (e.g., termination).
At Reflect, we are somewhere in the middle where we spin up browser sessions that are fronted by a load balancer (like a service), but each of them carries a ton of state and can fail independently. The end-user constantly manipulates the browser container, but the container also needs to send diagnostics and test results back to our internal service. It took us a while to effectively track the state of each browser session container from our management service, especially in determining all the ways that a browser session can fail.
Are There Any Invaluable Learnings During The Whole Product Development Process That You Would Like To Share With The Readers?
In terms of learnings, I think it may have been beneficial if we had approached the problem of managing browser session containers as their own service collectively. Even though we really don’t need a service fronting all session containers (since the end-user talks to a specific container directly), it would probably have implied at the outset some of the designs for container management that we ended up needing by the end. So, to summarize, before designing a highly distributed system, first consider the monolithic version of the service and learn from how that would behave to design the distributed version.
Is There Any Advice That You Have For The Devs & Entrepreneurs Who Aspire To Or Are Already Building Their Product From The Ground Up?
We benefited immensely from working on our idea, exploring technical feasibility and researching the existing market while we were still employed full-time at our previous jobs. In some ways, it can be difficult to find the motivation to work on a side project because you’re not dependent on its success financially and you’re probably exhausted after a full day of work. But if you can be disciplined in those moments when the pressure is not on, you will learn how to be disciplined when the pressure is on. It also gives you a running start when you leave.
Where Can We Learn More About You?
Our company site: https://reflect.run
The general information email address: [email protected]
My LinkedIn page: https://www.linkedin.com/in/fitz-nowlan
My personal site: http://root81.com
We love chatting about tech and entrepreneurship. And we’re happy to talk more about our infrastructure or DevOps in general. So, please get in touch!!
Lastly, I want to extend a personal Thank You to Shivang and the 8bitmen team for inviting us into this interview and letting us share our story.
Fitz, thank you for providing a detailed technical insight into your product & I wish you success as an entrepreneur.
If you guys are working on something cool or have already developed something & started up and would want to share your story. Check out this page.
Also, before you leave you might want to check out this Product Development Roadmap – Insights, By A Developer, Into The Process Of Developing New Products From The Bare Bones article that I recently published.
I am Shivang, the founder of this blog. You can read more about me here.
Here is my LinkedIn profile in case you want to say Hi!! 🙂
More On The Blog
- How Miguel Developed A Tool That Generates Static Versions Of Dynamic Websites
- The Co-Founder Of Reflect Shares Technical Insights Into His Novel Web-Based Testing Product
- The Co-Founder Of Symphonia Shares His Insights Into How He Developed A Social Music Composing App
- How Liyas Developed An Open Source API Testing Tool With Over 16K Stars On GitHub
- Product Development Roadmap – Insights, By A Developer, Into The Process Of Developing New Products From The Bare Bones