Short Bytes: Virtual personal assistant applications and digital home automation have become a hot topic with many big players placing their stakes in these futuristic technologies, but with the always-on microphones, telemetry, and phone-home function, these technologies can be just as scary as they are breathtaking. So, what do we do when the commercial options don’t fit our budget, privacy concerns, or security standards? Same thing we do every other time, we turn to open source.

Almost every major industry player in operating systems, both desktop and mobile, has cooked up a virtual personal assistant product, and then there are many other independent efforts as well. But how many open source virtual personal assistants are there? The sad answer is that there really aren’t many, but someone has made a large effort to increase the accessibility of this type of project and bring it into the open source community, so read on for our interview with Tanay Pant, author and maintainer of the virtual personal assistant Melissa.


We’ve already covered Mycroft, but it’s probably the best example. Mycroft is built upon the Raspberry Pi which is an open source platform itself, and additionally, Mycroft boasts an API designed to invite others to extend and add functionality to Mycroft. Mycroft is where the virtual assistant and automated home converge. The Mycroft team says that if it’s connected to the internet, the user can control it. With the open model and invitation for developers to extend Mycroft, there’s no telling how capable Mycroft will become over time. It’s best you keep your eyes on this one.


It would seem that the Raspberry Pi is a very popular choice for developers these days because Melissa is also designed to take advantage of the Raspberry Pi, but it’s compatible with other hardware and operating systems as well. Melissa is the brain-child of Tanay Pant which resulted from his disappointment in the barrier of entry in virtual assistant development. His inspiration was Jarvis of Iron Man, but his vision was for an extensible piece of software that could be customized by novices. He was determined, it seems, to remove the barrier he experienced himself. While Melissa isn’t a commercial product like Mycroft with commercial modules, it certainly has its own appeal.


Jasper is definitely much more DIY than the previous two. Jasper is an open source platform for developing voice-driven applications. While Jasper isn’t much of an assistant or automation tool in and of itself, the project does provide a handful of virtual assistant features to demonstrate the ability of Jasper.

An interview with Tanay Pant — Melissa’s inventor

As mentioned above, Tanay Pant, Melissa’s inventor and maintainer, kindly accepted an invitation for an interview and he answered a few questions about his experience with developing Melissa as well as some industry concerns.

What was the biggest barrier in entering virtual personal assistant development? Is it feasible for a novice programmer to develop his own?

The biggest barrier that I faced was that there is no definitive guide or tutorial in general for the beginners in the field. So, naturally, I had to sit back and contemplate on what the ideal workflow should be. The structure of Melissa has changed a lot over time and many contributors have joined my project now which is really wonderful. The problem with studying the code of the existing open source virtual assistants is that they are not designed to teach the craft to novices. After Melissa reached a stable point, I wrote a book titled Building a Virtual Assistant for Raspberry Pi which was published by Apress Media. The book assumes that the readers have a basic knowledge of Python and that’s it. It walks you through the whole workflow to the point that you can dive into our repository at GitHub and start working on a more advanced version of the project. So yes, it’s now feasible for a novice programmer to build their own virtual assistant.

What are the key components to a virtual personal assistant and how do they come together?


I like to abstract things, so I’ll say that any basic virtual assistant can be broken down into four essential components on a higher level. STT (for converting your speech to text to be processed by the logical engine), logical engine (for making sense of the text received by the STT and taking action), database (for storing the actions and commands by the user for later processing when you want to apply machine learning) and finally TTS (for responding to the user’s query). There are many good STTs and TTSs available which you can integrate with your project. So, it all boils down to the logical engine that you have to construct.

How do you think virtual personal assistants will change the future, if at all?

Yes, they most definitely will! And we are already moving towards that era with products like Alexa, Google Home, and Mycroft coming into existence. They allow a more natural interaction with computers and the automation work that they do, helps in reducing human effort which in turn increases our productivity. Remember the movie Iron Man? Having a voice controlled house automation of that scale is certainly something a lot of people are looking up to. Although, a system like Jarvis is very difficult to build with the currently existing technologies, the overall development environment and the tools/APIs that are being available are improving substantially to make this dream come true.

What are the pros and cons of open source vs proprietary projects of this kind?

Open Source projects follow a more community-based approach which allows different open source projects to learn and adapt from each other. Even if whole virtual assistants cannot be made open source by companies like Google and Apple, they should at least try to open source some of the core components. For instance, open sourcing TensorFlow was a good decision by Google. I feel that the core technology and commercialisation of the project are two entirely different things. Personally, I want the core technology that I have built to be open source so that other developers can build on it and they don’t have to reinvent the wheel. It’s better to open source your technologies because the community members provide insights and contributions that are really valuable. If it’s your first time open sourcing a software, you will be blown away by the benefits that you can reap out of an open source project as opposed to a proprietary project.

Are there any potential risks to using virtual personal assistants in terms of security or reliability?

Of course, there are numerous concerns! Security, reliability, and privacy can be potentially breached. I would say that for virtual assistants, with great power comes great responsibility. Hence, these systems need proper security auditing and users should always read the privacy policy of these systems carefully. It is the responsibility of the companies to take care of these concerns for the user.

Thank you, Tanay Pant, for your insightful interview about virtual personal assistants. We look forward to your work with Melissa and contributions to the open source community.

Have any Fossbytes readers read Tanay Pant’s book? Have you written your own virtual personal assist application? Let us know in the comments below.

Also Read: Top 15 Facebook Open Source Projects You Must Know