Agile: User Stories for Machine Learning Project? [closed]

Question

I've just finished up with a prototype implementation of a supervised learning algorithm, automatically assigning categorical tags to all the items in our company database (roughly 5 million items).

The results look good, and I've been given the go-ahead to plan the production implementation project.

I've done this kind of work before, so I know how the functional components of the software. I need a collection of web crawlers to fetch data. I need to extract features from the crawled documents. Those documents need to be segregated into a "training set" and a "classification set", and feature-vectors need to be extracted from each document. Those feature vectors are self-organized into clusters, and the clusters are passed through a series of rebalancing operations. Etc etc etc etc.

So I put together a plan, with about 30 unique development/deployment tasks, each with time estimates. The first stage of development -- ignoring some advanced features that we'd like to have in the long-term, but aren't high enough priority to make it into the development schedule yet -- is slated for about two months worth of work. (Keep in mind that I already have a working prototype, so the final implementation is significantly simpler than if the project was starting from scratch.)

My manager said the plan looked good to him, but he asked if I could reorganize the tasks into user stories, for a few reasons: (1) our project management software is totally organized around user stories; (2) all of our scheduling is based on fitting entire user stories into sprints, rather than individually scheduling tasks; (3) other teams -- like the web developers -- have made great use of agile methodologies, and they've benefited from modelling all the software features as user stories.

So I created a user story at the top level of the project:

As a user of the system, I want to search for items by category, so that I can easily find the most relevant items within a huge, complex database.

Or maybe a better top-level story for this feature would be:

As a content editor, I want to automatically create categorical designations for the items in our database, so that customers can easily find high-value data within our huge, complex database.

But that's not the real problem.

The tricky part, for me, is figuring out how to create subordinate user stories for the rest of the machine learning architecture.

Case in point... I know that the algorithm requires two major architectural subdivisions: (A) training, and (B) classification. And I know that the training portion of the architecture requires construction of a cluster-space.

All the Agile Development literature I've read seems to indicate that a user story should be the "smallest possible implementation that provides any business value". And that makes a lot of sense when designing a piece of end-user software. Start small, and then incrementally add value when users demand additional functionality.

But a cluster-space, in and of itself, provides zero business value. Nor does a crawler, or a feature-extractor. There's no business value (not for the end-user, or for any of the roles internal to the company) in a partial system. A trained cluster-space is only possible with the crawler and feature extractor, and only relevant if we also develop an accompanying classifier.

I suppose it would be possible to create user stories where the subordinate components of the system act as the users in the stories:

As a supervised-learning cluster-space construction routine, I want to consume data from a feature extractor, so that I can exist.

But that seems really weird. What benefit does it provide me as the developer (or our users, or any other stakeholders, for that matter) to model my user stories like that?

Although the main story can be easily divided along architectural-component boundaries (crawler, trainer, classifier, etc), I can't think of any useful decomposition from a user's perspective.

What do you guys think? How do you plan Agile user stories for sophisticated, indivisible, non-user-facing components?

Savory

Answer 1

It might be helpful to utilize the 'vertical slices' concept. Imagine a simple 3-layer application (UI/Logic/DB, for example). Instead of building out one layer, then another, you 'slice' vertically through all three. An initial story might be "As a user, I want to be able to log in to the system, so that I can access it." When done, this story is potentially-shippable, in that it provides complete functionality, but is highly unlikely to provide sufficient value to the customer to make it worth actually shipping. One benefit of vertical slices is that you've learned something about all the layers, knowledge which can be use in future iterations.

If you're not familiar with it, the INVEST model is very useful for User Stories:

I - Independent

N - Negotiable

V - Valuable

E - Estimable

S - Sized appropriately

T - Testable

Cappadocia

Thanks for the answer! I'm familiar with vertical slices, and I definitely agree that they provide a very nice model for designing and implementing isolated features in a GUI-centric application. But that kind of model doesn't work for designing a complex algorithmic data-oriented system. I can certainly divide the project along COMPONENT boundaries (there's a crawler, a feature-extractor, a cluster-creator, a classifier, etc. etc.), creating clear API boundaries between the different components. But this is not a layered application. There isn't even a user-interface. – Savory

How big is the Team working on this? How certain are you of the 'smoothness' of the journey? Is there value in putting this into 'real' user stories, aside from making the project mgrs and mgmt. happy? How long are your Sprints? – Cappadocia

Answer 2

Any story has a Role, an Action, and a Goal. So, think about writing a story which names an Role (a/k/a Actor) doing something to achieve a goal.

What you put down should have an obvious test, ie, an effective decision procedure that defines success and failure.

Where you're running into troubles here, I think, is getting caught up in "business value". Start by defining overall how you'll know when you'd completed your task successfully. Then, "achives business value" is making some progress toward the goal.

You have to be a tiny bit creative with some things in Agile, because they're so often oriented toward business processes.

Update:

Several points here.

1. it is a theorem that if you can't observe any effects of a component from outside the system, then that component can be removed without effect in the sense of observational equivalence.

2. There is defined a thing, usually called a task, that is a programmer assignment smaller than a user story. If you have something that seems to big as a story, break it up as a task. HOWEVER, do so in a way that it has well-defined external behavior, OR build it in a context in which you can observe its behavior.

So there are a couple of possible approaches that recommend themselves to me:

1. Set up big stories and break them into an unusually large number of substeps

2. Decompose the stories, perhaps by partitioning the data set. So, for example to decompose "User request tags updated", decompose your test data so that you have only data that would receive tag α and make a story "User requests tags updated to α". Since you know everything will be an α, you build the simplest code that always assigns alpha, and worry about the code that selects.

Openmouthed

With respect to "obvious tests", at the top level of the classifier, there are definitely some obvious tests, and I can already measure various different types of aggregate accuracy. But once I break the design into components, the tests become much less obvious. It's very difficult to test "feature extraction" in isolation from "classification" because the results of the classifier define the success criteria for the extracted features. No part of the system produces measurably correct or incorrect results, until those components are assembled into a complete system! – Savory

Answer 3

I think you can measure correct or incorrect results for a partial system as well. You need to stub out other system components. It is certainly possibly. Also, in my opinion it makes sense that one part of the system is an actor to other modules.

Mooneyham