How Visuality Systems overcame technical difficulties with the aim of developing proprietary CIFS/SMB solutions while helping bridge a geo-political conflict.
Think of an often encountered situation: What can one do when regional blocks can’t communicate well and are torn with turmoil?
In the late 1990s, whilst the tech world was riding on the internet wave and ushering in the digital sunrise, a starker situation prevailed in the Middle East, especially between Israel and Palestine. As the CEO of the Israeli branch of the giant telecommunication enterprise, Siemens, Sam Widerman thus sought to tackle both technology and the political developments in the region. His vision included doing his bit in resolving the Middle East crises and promoting the peace process between Israel and the Palestinians through joint efforts and financial incentives for both sides by opening additional offices in Ramallah and in Amman, Jordan with the full backing of the Siemens Headquarters based in Munich Germany.
Shaking up the scene
Sam left Siemens to found Visuality Systems in 1998 not only with the idea of nurturing the SOHO market with SMB solutions, but also to create a shared, collaborating and nurturing platform between Israel and the neighboring Jordan. The goal was clear – provide a workplace to Palestinians and let social benefits flow to their families. A high level of technical skills could surely bring the two sides closer.
The Server Message Block (or CIFS) space in those days was solely the domain of Microsoft. For Visuality Systems, the mission was to shake up the scene and bring this connectivity layer on top of every operating system, with a strong focus on Real Time Operating Systems (RTOS). SMB thus seemed to hold another meaning – Server of peace Messages for conflicting Blocks! The road however was not going to be easy.
Growth, setback and a phone call
In the initial years, the engineering team at Visuality Systems comprised 2 engineers from Israel and 14 from Jordan, the latter traveling across the border for training and development work. In three years, the company made significant progress with development work, whilst Jordanian engineers acquired skills to position themselves as top level engineers in Jordan.
Unfortunately, the outbreak of the second Intifada (Palestinian uprising) in September 2000 terminated the process along with the peace efforts. As fighting broke out between Israel and the Palestinians, Jordanian engineers had to stop their regular visits to Israel. The government had hardened entry norms, and there was an actual fear of the Jordanians being considered as spies. Visuality Systems was on the verge of shutting down, which would have happened but for a surprising phone call from HP. The rest was history…
Protocols are complicated
Undoubtedly, in-house development of SMB stack is a complex process. Apart from overcoming geo-political barriers, Visuality Systems had to face a number of obstacles in the first two decades of its existence:
- Lack of sufficient documentation: For years Visuality Systems worked hard doing reverse engineering, which took up time, required great creativity and intuition. In the beginning, it was difficult to follow Microsoft’s obscure specifications. The specifications today are clearer, but they are still far from being complete and need an open mind and creativity to implement.
- Protocol hierarchy: There are quite a number of protocols around SMB. It is relatively easy to develop SMB alone, but when a full package is required, it has to be developed with respect to the surrounding protocols (as shown in the scheme diagram below).
- Complexity: Simple calculations show that there are 19 SMB commands involved, but this is far from the reality. The CREATE command itself has 15 “contexts”, each of them being a type of sub-command. The IOCTL command should implement in at least 15 “file functions”, each of them representing another level of sub-command. This is without the file system functions. The SESSION_SETUP command also drills down into several sub-protocols. This adds up when considering QUERY_INFO, SET_INFO and other commands. Though not all levels were counted, the number of individual sub-sub-sub commands is estimated at 200.
Furthermore, it is not enough to develop an SMB syntax alone, because SMB semantics is also a must. The complexity of SMB semantic is levels above the complexity of SMB syntax. For instance, the behavior of an SMB server on receiving the CREATE command depends on quite a number of factors: File status, other client connections, user access rights, and many more.
2. High investment: The ratio between writing code and improving the code is 10 to 1, if not more. Visuality Systems considers the following tasks as code improving initiatives:
- Polishing: It is not enough to code an SMB server. It also needs excellence. This means that the solution must provide file sharing for tens of different operating systems as well as for hundreds of different applications. This system is multi-dimensional. For instance, Excel running on Windows 7 behaves differently from Excel running on Windows 10. Even worse, the same Excel runs differently on the 64 and 32 versions of the same Windows. Finally, Excel 2007 is different from Excel 2010. This is just one example, and there are hundreds of different applications today in the market, each one producing a different SMB traffic which the SMB server should be compatible with.
- Bug fixes: Commonly, bugs are revealed by both QA and in the field. However, contrary to “more convenient” products, the share of issues detected after unit testing is much higher when it comes to SMB. This is because SMB traffic has many degrees of freedom which generate endless situations. Developers can only cover some of them during unit tests. QA covers more, but many bugs still remain uncovered and these pop-up in the field. When it comes to SMB development, support is always an expensive matter.
- Monitoring the specs: As mentioned earlier, an SMB server is not merely the SMB protocol, and there are many protocols to deal with. Each of them has respective specs governed by Microsoft and/or other authorities. There are around 20 relevant documents. Typically, modifications are applied to each document at least twice a year. This means that an SMB developer will need to consider software modifications every second week. The best way to go here is to accumulate new specification features and to release the changes twice a year. Of course, a customer may require the new features immediately. This is not always justified, but since the customer is always right, a solution must to be found. In this case, there is no choice but to release a “hot fix”.
For the Windowless world
As Visuality Systems proceeded on its mission, two products have evolved from its offices: YNQ™ and NQ™ Storage into a worldwide success story. YNQ™ brings portability with SMB server/client architecture for non-Windows embedded systems to network with Windows-based environments. Uniquely flexible and easy to implement, it can be integrated with any operating system, any CPU, and basically with just about any embedded device. The embedded system can hook into any Windows network and carry out file sharing activities.
NQ™ Storage, a highly portable and scalable SMB solution, is meant for data centers and storage vendors who operate with non-Windows platforms. It allows them an SMB connectivity to any Windows-based environment. With the growth in data centers and their importance, a solution like NQ™ Storage can boost performance for a wide range of storage services, from standalone NAS to high-end data centers.
Today, Visuality Systems prides itself on close to 20 years of SMB experience, and is considered the world leader in the commercial SMB space.
We would love to hear from you and address any question or query you may have about Visuality Systems or its products and services. Please contact us at: [email protected]
Tal Widerman, CEO, Visuality Systems
