Jidoka
Way of Working
Overview
Jidoka is a software development methodology that aims to increase the automation and quality of a software project through the use of models and pipelines.
By following Jidoka, organisations can continuously modernise their software systems at scale while avoiding many of the problems associated with legacy systems.
Our hypothesis is that Jidoka is well-suited for large scale modernisation projects where the benefits of automation are most acutely felt. Some of these benefits include:
- Consistent architecture with low technical debt
- High test coverage including UAT's
- Lower organisational change with coarse grained milestones
- Decreased costs over the entire project lifecycle
- Up-to-date user and developer documentation
- Decreased knowledge loss and turnover
- Increased cross-team collaboration
- Healthier team culture and momentum
- Increasing ROI for modernisation portfolios
History
The software development industry has taken much inspiration from lean manufacturing. It is worth knowing as Agile, Scrum, Kanban, DevOps, and others have roots in this history. For example, TPS is a well-known implementation in lean manufacturing.
The Toyota Production System (TPS) was established based on ... "jidoka" (which can be loosely translated as "automation with a human touch") ... human engineers meticulously build each new line component by hand to exacting standards, then, through incremental kaizen (continuous improvement), steadily simplify its operations. Read more about Toyota Production System"
While there has been many different interpretations of lean manufacturing into software development methodologies, our interpretation focuses on brining human and machine skills together to produce a result that neither could reach alone (read more on centaur chess teams).
Up Next
Comparison
Learn about our process and solutions:
Comparison
Tackling software development
Comparing methodologies
There are many well-known and published software development methodologies and other approaches to problem solving. Jidoka can be uniquely applied to projects as it has advantages others didn't have available when they were conceived. The advantages have a central theme of automation:
- Years of R&D investment into our sister company Codebots.
- Model-Driven Engineering (MDE) allows us to build models (of almost anything) and then do useful things with those models.
- Pipelines can help us build software in a repeatable and scalable way using DevOps and modern technologies like Docker, K8s, etc.
- Knowledge management in its simplest form using markdown stored in a code repository.
- Recent real world experience of building 100+ projects and experimenting with different methodologies to find what works and doesn't work.
So, with these advantages in mind, let's examine some common acronyms and well-known development methodologies.
Up Next
Approaches to problem solving
Learn about our process and solutions:
Comparison
Tackling software development
Approaches to problem solving
The following three acronyms are all different approaches to problem solving and finding a solution:
- Proof of Concepts (PoCs) are great for discovery and risk mitigation within a narrow problem set.
- Minimal Viable Products (MVPs) are great for iterating through a number of build-measure-learn cycles when the solution is not known.
- Shortest Path to Value (SPV) is great for the partial replacement of a legacy system.
While each of these are great under some circumstances, when undergoing a large scale modernisation project, there are some significant drawbacks:
- PoCs are ultimately thrown away as their functional and non-functional requirements are too narrow to fit in a large scale modernisation.
- MVPs are searching for a solution but in large scale modernisations the target is known making the search less important.
- SPVs are designed to work in large scale modernisation projects but are used in conjunction with the strangler fig pattern that takes way too long.
How will Jidoka learn from these?
- PoCs will be used for tech spikes to remove risk from a project as early as practical in the project. The code from the PoC will be used as a reference for upgrading a bots capabilities.
- MVPs will not be used as we will replace the legacy system as close to like-for-like (L4L) as practical. We will lean towards deferring improvements to the legacy system until after the replacement system is on the new technology stack and in a far better position to consider changes.
- SPVs will not be used as the integration complexity of a project increases as functionality is replaced (strangled) from the legacy system. We will consider larger, most times the whole, legacy system to be replaced.
As crazy as that last point sounds, we are advocating a big bang approach. If a big bang approach can be objectively analysed, there are some significant benefits that could result. Here are some points to consider in favour of a big bang:
- Each time a legacy system is divided (or strangled) many integrations are created between the old and the new systems. This boundary is slow to form and adds more risk. By minimising the number of integrations, overall time in the project can be saved.
- Traditional software teams (without a codebot) had to fulfil requirements by manually writing code. For these teams, a big bang would take far too long before they shipped something due to the manual work, making the project infeasible. A codebot circumvents this problem as many requirements can be fulfilled at scale.
- Under circumstances where the solution is not known, a big bang approach is not recommended because dividing the work into smaller parts gives more opportunities to explore and test assumptions (like for an MVP). In a modernisation project, the solution is known so the need to divide the work and explore, test assumptions, etc is far less.
- Taking longer by unnecessarily dividing a legacy system will expose an organisation to getting caught part way through the modernisation if the project is halted for external reasons (like budget reprioritisation). In other words, shorten the length of time so you don't get caught part way.
Will a big bang be possible on every project? Most likely not, you may need to do a few smaller fire crackers but do as few as you can. Play to your strengths.
Up Next
Software development methologies
Learn about our process and solutions:
Comparison
Tackling software development
Software development methodologies
There are many different approaches to developing software and lots of talk about which one is the best software development methodology. Being a student of our industry, we have experimented with many and come to recognise there is a meta-methodology that could be used to describe them all. So, for Jidoka, it is important to set a few parameters and specify what parts of these other methodologies we keep, and what parts we leave behind.
| Methodology | What do we take? | What do we leave behind? | Summary |
|---|---|---|---|
| Scrum | Huddles Ceremonies Backlog Definitions | Sprints Story points Velocity | The Scrum meeting pulse (huddles, ceremonies, etc) is amazing for communication and the attention to backlog refinement raises project quality. Constant sprinting is tiring and story points are a waste as they get converted to time anyway. Velocity does not consider mass (size) of the system, momentum does. |
| Kanban | Kanban board Checklists Continuous flow | WIP limits Specialised teams Handover loss | The continuous flow of work across a Kanban board can far outpace other approaches that are stop/starting. However, the balanced queues of WIP limits lead to specialised teams with all the problems of waterfall reemerging. |
| DevOps | Pipelines Removing barriers | Rules complexity Burdensome governance | Breaking down the barriers between development and operations using pipelines to automate and scale is empowering. But some practitioners use this to enforce overly complex company policies and can rebuild that wall that was originally torn down. |
Up Next
Project types
Learn about our process and solutions:
Project types
Unpacking the problem
Analysing applicability across diverse project types
To focus the methodology on only some projects, we have grouped projects into three different types. Even though Jidoka has application in a wider scope, our hypothesis is that the benefits of automation with a human touch would be most recognised in these types of projects.
Up Next
Like-for-like
Learn about our process and solutions:
Project types
Unpacking the problem
Like-for-like
One type of modernisation project is a like-for-like (L4L) replacement of the legacy system.
The solution is known as there is a system already in place.
The legacy system has become a liability to the organisation so there is a need to replace it.
The initial replacement of the legacy system should avoid changing any organisational processes where possible.
Avoiding organisational change will increase the likelihood of a successful modernisation.
Retraining of staff will be minimised as the new system will be familiar.
Changes to the new system should only be considered after the replacement is completed.
These new requests will be satisfied far more efficiently given the new technology stack and environment.
Up Next
Spreadsheets
Learn about our process and solutions:
Project types
Unpacking the problem
Spreadsheets
Spreadsheets are a common approach for organisations to use as they have a low barrier to entry.
- The solution has been developed into a spreadsheet so many of the complexities around the data and its formulas have been solved.
- The spreadsheet has become a liability to the organisation as it cannot scale due to spreadsheet limitations.
- The initial replacement of the spreadsheet should avoid changing any organisational processes where possible.
- Organisational change around modernising a spreadsheet compared to a L4L replacement is a higher risk.
Up Next
Disparate systems
Learn about our process and solutions:
Project types
Unpacking the problem
Disparate systems
Disparate systems are common as organisations organically grow over time. The systems can be digital or manual systems and have been built without considering the whole.
- The solution has been built piecemeal so that some of the complex problems have been solved.
- The disparate systems have become a liability to the organisation as it cannot scale due to the lack of cohesion.
- The replacement of the disparate systems will likely incur changes to organisation processes as multiple systems are considered.
- Organisational change must be carefully managed.
- Retraining of staff across the organisation must include regular communication channels and feedback mechanisms.
- Changes to the new system must be considered meticulously as the downstream effects will slow the momentum of the project.
- If changes can be deferred, these new requests will be satisfied far more efficiently given the new technology stack and environments.
Up Next
Process
Learn about our process and solutions:
Process
Vision = reality
Process
Over a series of milestones, the team diverges and converges on goals by following a set of principles to deliver both functional and non-functional requirements. There are three stages to the process (each consisting of one or more milestones) that carefully consider the current state of the project:
- Discovery: In this stage, we lay the foundations for success and formulate a plan that the whole team is confident will work.
- Modernisation: In this stage, we execute the plan and play to our strengths by modernising the legacy system with as little change management as practical.
- Optimisation: In this stage, we ensure the system is monitored and remains secure, while incremental improvements are now unblocked as we are now on the modernised technology stack.
Up Next
Stages
Learn about our process and solutions:
Process
Vision = reality
Stages
Some important dot points that highlight a stage are:
Discovery
- Diverge to converge on a plan.
- Use the activity kit to build knowledge.
- Set governance and standards.
- Tech spike high-risk issues.
- Establish a backlog of functional requirements.
- Set goalposts for non-functional requirements.
- Promote, fork, and merge a bot plan.
- Create a communication plan with the user base.
- Complete at least one milestone.
Modernisation
- Execute the plan with one or more milestones.
- Harden the target environments and monitor.
- Emphasise the data-driven pipeline reports.
- Communicate milestones to users.
- Adhere to governance and standards.
Optimisation
- The DevOps checklist is the standard that must be kept.
- Keep pipelines green.
- Monitor the target environments.
- Bug fixes.
- Upgrades and patches.
- Make improvements.
- Open-ended milestones without an end date.
- Adhere to governance and standards.
Up Next
Milestones
Learn about our process and solutions:
Process
Vision = reality
Milestones
- Each stage can have many milestones.
- Milestones have goals with date ranges and we aim for the closest date.
- Milestones are not fixed length like sprints.
- If the squad feels it is necessary, they may break a milestone down into a number of iterations (of variable length), but this is not always the case.
- A kanban-style continuous flow of issues is worked on throughout the milestone until the goal is reached.
- Ceremonies, meetings, checklists, definitions, etc are all performed according to the plan.
- The depth of details in an issue is proportional to the issue's risk
- We plan for breadth-first rather than a depth-first approach by taking advantage of automation, modelling and pipelines (read more below).
Most problem-solving approaches use a depth-first search, i.e. a depth-first search starts at the root node and explores down the branch that shows the most promise and then backtracks if no solution is found. In a way, this is how MVPs and SPVs work. They approach the problem incrementally as the solution is not known.
Another approach is a breadth-first search, i.e. breadth-first search starts at the root node and explores all the nodes at the present depth before moving on to the next. In this way, we can approach a L4L replacement as the legacy system is known and we are not searching for something we don't know. Identifying this as a key difference presents an opportunity for how we approach a milestone.
By considering the whole legacy system for replacement, unlike the strangler fig approach of SPVs, we will have a better understanding of the gaps across the entire system, not just a narrow part. In a milestone, we:
- Model and generate as much of the legacy system as possible.
- Analyse the gap and what requirements are not being satisfied.
- Look for patterns in the gaps that are common.
- Extend the bot's capabilities to fill the biggest gaps.
- Anything one-off, we hand code.
- Start this process all over again (e.g., go back to 1).
Up Next
Estimations
Learn about our process and solutions:
Process
Vision = reality
Estimations
There are many ways to estimate a software project and we have experimented with many. What we have learned is that it is very time consuming and can create over-estimations when using issues to estimate as they are too fine grained. On the otherhand, we don't want to use #NoEstimates as it has too much of a negative impact on expectation management. With this in mind, the high-level, balanced approach to estimations is:
- A milestone has a goal and the team uses MoSCoW (must-have, should-have, could-have, won't-have) to prioritise requirements.
- Estimations are performed at the milestone level and not the issue-level.
- The team must agree on a realistic amount of time they need to complete the milestone.
- The team doesn't stop at the must-haves and continuously works Kanban-style on the backlog with the available time.
- The squad lead and account manager will calculate an appropriate buffer based on the risk associated with the milestone and communicate this date to the product owner.
- The team still aims for the closest date.
- Nothing builds trust faster than finishing earlier and delighting a customer.
Lastly, the account manager will be in charge of the estimations throughout the tender or bidding process so the team will inherit an overall time for Stages One and Two. It is imperative that the account manager consults senior team members to gain advice on how big the project should be so there is enough time to ensure a high quality project.
- 8 to 12 weeks for Stage One.
- 3 to 8 weeks for each milestone in Stage Two.
- 15 to 40 weeks for five milestones over Stage Two.
Up Next
Principles
Learn about our process and solutions:
Principles
What guides us
The shining light that guides projects
Last, and definitely not least, we can now unveil the principles that will be used to guide Jidoka projects:
- Augment the intelligence of a centaur software team with a codebot.
- Treat everything as a model and do useful things with them at scale.
- Knowledge is understanding and must be recorded in its simplest form in a code repository.
- Pipelines lead to quality, increased automation, and so many other benefits.
- Address high risk issues early and take on the most challenging first.
- The admin associated with something should be proportional to its risk.
- Ignore non-functional requirements at your peril.
- Avoid changes to the organisation's business processes while modernising, leave it to optimising later.
- Plan to be continuously modernising and keep momentum.
- Find balance and only use these principles where practical...
Up Next
Next steps
Learn about our process and solutions:
Next Steps
Theory is great but that’s not why we created the Way of Working. This is a practical guide to helping you deliver better software projects. There are two paths you could take as logical next steps.
Path 1: Internal Adoption
The first path you could take is to use the Way of Working and implement it internally. This means training your team on the process and customising it to suit your company and delivery preferences.
Start by embedding some of the ceremonies into your teams everyday routine. Start the day with a huddle, ensure that every iteration is planned and reviewed and that there is regular user acceptance testing. Developing a pattern and cadence for these ceremonies will start embedding the Way of Working into your process.
If you are struggling with implementing the Way of Working, please reach out. We are always happy to go for a coffee.
Path 2: External Understanding
The second path you could take is to learn the process intimately but not necessarily adopt it yourself. We would recommend this pathway if you’re a product owner that is looking to engage a software development company in the future, or if you’re presently in the middle of a project.
While most software development companies should have a Way of Working, this is not always the case. To complicate things further, the level of detail in one company's process may not be the same for others in the industry. It’s important for product owners that are part of the business funding the project to have that background context of what a robust process looks like.
Most activities in the Way of Working involve the participation of a product owner. Understanding your time commitments throughout an application’s lifecycle can be incredibly valuable. Whether you engage WorkingMouse, or another software development company we would recommend gaining an in-depth understanding of the Way of Working.
Learn about our process and solutions:
