Assessing a Technical Organization Questions

General Questions

  • What’s the culture like?
  • Are developers on call?
  • How often do managers take vacations?
  • Tester question:  You turn in a bug to a developer and they come back with ‘it works on my machine’ what does the tester do?

How They Work

  • What software development practices does your team follow? (Agile, Waterfall, etc)
  • How long are the Sprints if agile?
  • How would you describe working in an agile culture? (Helpful, Overbearing, Waste of time)
  • How many products does your company/team develop?

Technical Questions

  • What is the current architecture of your current products?
  • How often does the team release software?
  • What does the release process look like? (this may be different for different products)
  • Do code reviews happen? How often and by whom?
  • What does your use for Source Control?
  • Who all has access to the code?  Testers?  Devs on other teams?
  • What does your code branching strategy look like?
  • What does your test environments look like?
  • What does the release process look like? (Release Management)
  • How, when, and where is the software tested?
  • When does the test team get involved in the SDLC?
  • What tools do the testers currently use?
  • Do you have access to the code changed on a per test item basis?
  • How does your team manage defects?
  • How do you know what to test?
  • Has there been any performance testing done?  Is it needed?
  • What type of security testing has been done?
  • What’s the process if there is a critical bug released to production?
  • Is there any test automation?  Unit, Integration, UI, API, Smoke?
  • What are some of the ways you mitigate risks prior to going to production?
  • How close are Test and Production environments?
  • Is there Test Environment or Production environment monitoring, if so to what extent?
  • How are critical bugs in production discovered? (monitoring, customer calls, post release testing?)
  • What is the relationship between the Developers and Testers? (do they sit together, plan together, collaborate together?)
  • What’s been the test strategy around different browsers?
  • What’s been the test strategy around mobile device (app/responsive) testing?
  • Is there any external Hardware decencies that your product supports?
  • What is the most complex area of the software system?
  • Is there good documentation about the product? - Great place to start documentation
  • Are there are 3rd party integrations?  Payments, Reporting, API inbound/outbound?

Product Questions

  • What are the most important aspects of Club Automation? What makes it unique?
  • What areas in the product are most widely used by our customers? What type of things would make our users angry?
  • What’s the current market focus?
  • How are we better than our competitors?
  • What areas are problematic that our competitors beat us in?
  • What is the biggest risk in your opinion?

Personal Questions (1:1s)

  • How’s life?
  • What have you been excited about lately?
  • Anything have you worried or down lately?
  • What are your expectations of me as a leader?
  • What do you love doing right now?
  • What do you loath doing right now?
  • What area do you feel the weakest in?
  • What area do you feel the strongest in?
  • What accomplishment are you most proud of over the past quarter?
  • What are you most excited about at Daxko/CA?
  • Where do you see yourself in 5 years?
  • What does success look like for your career 3 months, 6 months, 1 year from now?
  • If you could be proud of one accomplishment between now and next year, what would it be?
  • How is your current workload?
  • How can I help you succeed this week?

Empowerment Questions (1:1s)

  • If you were leading this team what would you do different?
  • What are some things we are not doing that we should be doing?
  • What could we be doing different/better? *Is there anything that your team is working on that isn’t providing value to Daxko?
  • Is there anything that you are currently doing that isn’t pushing Daxko forward?
  • How do you think ________ went?

Clarity Questions (1:1s)

  • Do you have questions about, direction, strategy, major initiatives, progress?
  • Do you feel like you’re getting enough feedback on your work? If not where would you like more feedback?
  • What are you least clear about in terms of our strategy and goals?
  • What area of the company would you like to learn more about?
  • Is there anything you’d like to know about the market we serve?

New Reportee Questions (1:1s)

  • What are your current responsibilities?
  • What interests you the most these days?
  • What have you come to know you don’t want to do?
  • What have you been told about this opportunity?
  • Things I think I can help him\her out with?
  • Things I think I would forsee him\her potentially helping or owning
  • Is there anything else you would like me to know about you?
  • Is there anything you would like to know about me?
  • During what part of the day do you focus best?
  • Setup a 1:1 cadence during the first conversation.

Questions to ask Leadership

  • What are your expectations for me?
  • What is the biggest risks?
  • How often can we check-in? 1:1

More Good questions (Give 1 point for each ‘yes’)

  • If you ask the team to change their codebase do they react positively?
  • Does each member of the team have access to the system source control?
  • Does the team know which parts of the codebase are subject to the most change?
  • Does the team collaborate regularly with teams that maintain their dependencies?
  • Does the team have regular contact with the users of the system?
  • Can you set your system into a given state to repeat a test?
  • Is each member of the team able to create a disposable test environment?
  • Is each member of the team able to run automated unit tests?
  • Can the team test both the synchronous and asynchronous parts of their system?
  • Does each member of the team have a method of consuming the application logs from Production?
  • Does the team know what value the 95th percentile of response times is for their system?
  • Does the team curate a living knowledge base about the system it maintains?

Notes On Each Question

  1. If everyone actively recoils from change to all or a specific part of the system and there’s always a reason not to tackle this. It’s a no.
  2. Everybody should be able to see the code. More importantly a lot of collaboration occurs here. Access for all is part of closeness as a team.
  3. A codebase generally has hotspots of change. Knowing this can inform what to test, how to test and where observation and control points are.
  4. Teams have both internal and external dependencies. The testability of dependencies effects your testability. Close collaboration such as attending their sprint planning helps.
  5. To test a system effectively one must have deep empathy with those who use it. For example if you have internal users you invite them to demo’s, for external customers your team is exposed to customer contacts, or dare I say it, even respond to contacts.
  6. Some tests need data and configuration to be set. Can you reliably set the state of these assets to repeat a test again?
  7. Sometimes programmers have environments (I hope) to make their changes in something representative of the world. Your testers and operations focused people should have this too to encourage early testing, reduce dependency on centralised environments and setup time of tests.
  8. If you don’t have unit tests, you probably haven’t even tried to guess what a unit of your code is. Immediate no. If you do each member should be able to run these tests when they need them.
  9. Systems often have scheduled tasks which lifts and shifts or replicates data and other operations which occur on demand. Can you test both?
  10. Its important that all teams members can see below the application, so we aren’t fooled by what we see. If you don’t have an organised logging library which describes system events you should get one, plus eventually some form of centralisation and aggregation. Or just use syslog.
  11. If you don’t know this you either haven’t been able to performance test, gather data for analysis, or realised that you can analyse your performance data without outliers. Either way you probably know little about what your code does under load or where to start with tuning.
  12. Documentation. I know you are agile and you thought it was part of your waterfall past. To be fair, this is anything that describes your system, a wiki, auto generated docs, anything. If someone wants to know what something does, where do you go? Only the source code alone is a no for this one. Your team has many members, not that many Product Owners can read code.

If = 12 -  You are the masters of observation, control and understanding. Change embraced openly. Way to be. If = 11 - Danny Dainton’s Testability Focused Team at NewVoiceMedia - Wednesday 2nd August 2017 If = 10-8 - You are doing pretty good. You can change stuff with a reasonable degree of confidence. Could be better. If less than 8 - Uh oh. You are missing some big stuff and harbouring serious unseen, unknowable risk. Must do better.

The Joel Test: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ 

1.  Do you use source control? 2.  Can you make a build in one step? 3.  Do you make daily builds? 4.  Do you have a bug database? 5.  Do you fix bugs before writing new code? 6.  Do you have an up-to-date schedule? 7.  Do you have a spec? 8.  Do programmers have quiet working conditions? 9.  Do you use the best tools money can buy? 10.  Do you have testers? 11.  Do new candidates write code during their interview? 12.  Do you do hallway usability testing? The neat thing about The Joel Test is that it’s easy to get a quick yes or no to each question. You don’t have to figure out lines-of-code-per-day or average-bugs-per-inflection-point. Give your team 1 point for each “yes” answer. The bummer about The Joel Test is that you really shouldn’t use it to make sure that your nuclear power plant software is safe. A score of 12 is perfect, 11 is tolerable, but 10 or lower and you’ve got serious problems. The truth is that most software organizations are running with a score of 2 or 3, and they need serious help, because companies like Microsoft run at 12 full-time. Of course, these are not the only factors that determine success or failure: in particular, if you have a great software team working on a product that nobody wants, well, people aren’t going to want it. And it’s possible to imagine a team of “gunslingers” that doesn’t do any of this stuff that still manages to produce incredible software that changes the world. But, all else being equal, if you get these 12 things right, you’ll have a disciplined team that can consistently deliver.

https://www.maxongzb.com/choosing-the-right-startup-read-time-3-mins/

Right questions to ask when assessing a startup

https://twitter.com/noahsussman/status/1130117747034284034?s=11

fwiw one of the first things I do with a client is get their list of “customer support workarounds” and prioritize fixing all of the issues, then making it possible for CS to report further issues directly to eng. The cost of CS workarounds is ignored at the org’s peril.

Test Automation Framework Guidelines

  • Is it running at least nightly?
  • Is it running in the cloud or on a server?
  • Can it run on anyone’s machine?
  • Is there a good readme to get folks up to speed on the framework
    • How to run
    • Links to libraries used and their documentation
    • List of special dependencies
  • Is it in a repository/source control?
  • Is Test Data Strategy documented?

Nice to Have

  • Additional documentation on how to write specs
  • Training Videos where new joiners can get ramped up on the framework

Due Diligence External