Enterprise Architecture Analysis in a Graph Database

Abstract

For a long time I have wanted to be able to get data out of the excellent Archi tool into a place where I can perform meaningful queries on it. Queries such as how are items connected, which  are the most connected and so on? 

This blog describes the journey I went through, what I discovered and how I achieved what I wanted, and more.

The Tools and the Journey
Archi
The Archi modelling tool supports latest ArchiMate standard and now has a plugin for several fellow Enterprise Architects to contribute to the same model. I also employ the forms plugin (to easily enter data and export to Excel for some pretty graphs), along with the Specialization plugin, to allow me to decorate some of the elements in new ways.

Jasper
I could use the Jasper reporting tool which the Archi team have integrated as their Reporting tool of preference, but that was so complicated as to put me off trying - even with tweaks to the example reports I felt it wouldn't give me what I needed anyway (I have a large archimate model with my latest client).

Open Exchange File (OEF)
Archi exports to Open Exchange File format. From there I can use some funky XLST to transform the data. One suggestion from (ref) was to export to GraphML in order to import into Neo4J. 

GraphML
Is a viewing tool for graph based data. This blog (ref) was a source of inspiration to using OEF /  XLST manipulation in order to create a GraphML file. I even got the XLST files from (ref) although he was kind enough to say he hadn't touched them in a year. The OEF standard that Archi exports had changed in that time, but it was easy to map over. I had created a .graphml file and could open it up. I couldn't easily see how to load it into Neo4J.

Gephi
I even looked at Gephi, and whilst it had some great visualisation functionality, I couldn't do my queries. What I still needed was a database of some kind.

CSV Export
Having gone down a hole with XLST transformations, I wanted life to be easier. Good old comma separated value exports are easily done from Archi, out of the box. This gives you three files: a list of elements, a list of the relations between them and a list of properties that were defined on the elements or relations. There is a good discussed about using CSVs to get p

MySQL
I'm an ex Relational Database geek so I withdrew to familiar territory. My MacBook easily installed a MySQL database and the CSV import from elements and relations was a breeze. Cross-queries to between elements and relations allowed me to get a good level of insight. But I realised I could get navigation and connectivity from proper use of native Archi using its excellent visualiser or navigator panes. So, I bit the bullet and went back to a proper graph database that I'd downloaded a week before (when creating the .graphml files) and hadn't used.

Neo4J
I found the latest Neo4J Desktop distribution easy to install and setup. The manual and various YouTube  videos showed CSV IMPORT as the way to go. The basic form of CSV import was alright, but I wanted to label the nodes with the element types, label the relationship with the relationship type and I also wanted to add properties to the elements. Basic CSV IMPORT doesn't allow parameterised labelling. 
I stepped up the experience a level. I discovered and installed the APOC library (it's handily available in the plugins area of the Neo4j Desktop databases). This gives you around 300 power procedures, including the ability to import CSV in the way I wanted. 

JArchi
This is the JavaScript language that is available from Archi through Patreon support. I'd urge anyone to give the team some support and to have a look at this. JArchi is a really flexible way of getting at the data (and updating it). It would be easy to get the data you need in a CSV in order to get it into Neo4J. (However - there's a better way; see later). Here is more information on JArchi.

Archi Tool Visualiser 
The built-in Archi Tool Visualiser is itself a graph way of looking at what's connected to what. It's an easily overlooked facility and now that it also has a bunch of filters it makes it possible to ignore / include relationship types, to certain depths and so on. Rush to page 70 of the user manual for that one.

The Interim Outcome
Before I discovered the Archi Database Plugin (below), I had to use three command Cypher statements to load of all the data from the three Archi CSVs (To create labelled element nodes, labelled relationships and then to associate the properties against the elements).
That's the minimum as you also need to create a number of indexes using the uuid's. 

Looking at my data in a new way highlighted some inconsistency in my Archi model. I had null properties. It also tripped over descriptions with double quotes in (there's probably a simple way to fix that?).

I also have properties on my relationships, so I need a nice way of deciding whether to link them to the element nodes or relationships (apart from listing the object type - temporary fix).

New Enlightenment
Since my foray into this, I received an email from Thomas who I had kept up to date with. He told me there had been a lot of activity on this topic recently and linked me to an architoolkit tweet which pointed me at

https://lju-lazarevic.github.io/ArchiMateNeo4j1.html

This was exactly what I was trying to achieve and whilst pipped to the (blog) post I enjoyed its thoroughness.

The Right Outcome
Archi Database Plugin
This amazing database plugin allows us to export and import models via a SQL database (Microsoft SQL Server, MySQL, Oracle, PostGreSQL, and SQLite are supported so far), plus the ability to export elements and relationships to Neo4j. Yes! In fact the ability to just re-export the model into Neo4J to overwrite it makes doing the refresh easy.

The Queries
I initially wonder why it is worth using Neo4J to analyse the models, especially when you can do it in JArchi or via a (simpler?) SQL database. However, it is just more dynamic and visual, of course. It really works well with highly complex data structures (which a mature Architecture repository based in ArchiMate becomes). It helps 'see the woods for the trees' and leads you into new 'what if' queries and dependency queries that are not naturally possible with scripts and SQLs; for example:

• Element dependencies
  
• Unconnected elements
  
• Elements that don't connect throughout all the domains
  
• List the Value and Applications for a Business Capability
  
• Listing Applications that are sunsetting or retiring
• Identifying corners of the architecture that are complex and maybe shouldn't be
  

What's Next
I'd really be interest to be able to use some immersive technology such as 3D gaming glasses to investigate an ArchiMate model. Imaging wearing the goggles and just pointing across an architecture? How to link Neo4J databases to this technology needs more time that I currently have, but I'm convinced it is possible.