Boolean Search String Experiment Follow Up

On November 8th, 2010, I wrote a post containing a Boolean search challenge and an experiment of sorts – I asked readers to share their approach and Boolean search strings for a basic job description. The inspiration for the experiment came from the fact that very few people seem to be consciously aware of the issue that when it comes to sourcing candidates via the Internet, resume databases, LinkedIn, etc., is that all Boolean candidate searches work, provided they are syntactically correct.

This is a fundamental problem which heavily influences the perception of sourcing as a low level, non-critical function and/or role, because anyone can take the title from a job description and the required skill terms, create a basic Boolean query, and get results. This leads to the idea that finding talent is easy – slap a few search terms together and voila! – you get candidates.

Congratulations for finding the same candidates everyone else is finding with the same unsophisticated searches. All candidate queries are definitely not created equal, and you simply cannot gain any competitive advantage running the same basic taken-straight-from-the-job-description title and keyword searches that everyone else does.

The lesser-known reality is that most people who run Boolean searches on LinkedIn, job board resume databases, in their Applicant Tracking Systems (if they even support Boolean – ouch!) and the Internet only find a small fraction of the talent that is available to be found. I’ve written quite a bit on the topic so I won’t belabor that point in this post.

The Boolean Search String Experiment Succeeded AND Failed

Unfortunately, in many organizations, sourcers and recruiters do not get (or seek out) the opportunity to compare and contrast their search strategies and tactics with their peers and/or managers on a position-by-position basis. Much of the magic of using Boolean queries for talent discovery and identification, or lack thereof, happens on each person’s computer screen and thus a great number of people have absolutely no public basis of comparison.

As someone who has trained several hundred recruiters and sourcers, I know that even for basic positions and hiring requirements, 10 different recruiters/sourcers will come up with 10 different search approaches, and while they will find some of the same people, each is likely to find some that none of the others will.

I wanted to show the world that point – to give people a public basis of comparison.

To date, the Boolean search string experiment has had 1,106 unique views with an average “time on page” of over 5 minutes, which is pretty solid for a short post. I was sincerely hoping I would get at least 100 responses to the Boolean search string experiment.

So far, only 36 intrepid people (thank you!) have shared their approach to analyzing the job description I used for the experiment and the Boolean search strings they would use to try and find qualified candidates – a whopping 3% of the number of people who have viewed the post. It’s also worth noting that 9 of the 36 work for the same company. :-)

As I and a few people have pointed out – there is no single best Boolean string or approach. However, I did see a number of  intelligently and logically constructed Boolean strings and well thought out analyses and search strategies.

Why the Low Response Rate?

I have several ideas as to why there were so few responses.

I’m only going to share one of them – I think most sourcers and recruiters aren’t truly confident in their Boolean search string skills.

I’ll put it this way – nearly everyone is comfortable dancing and singing when no one is looking and they are alone, but that all changes in a public forum.

However, if you know you’re a really good dancer or singer, most people have no problems dancing or singing in front of others. In fact – many enjoy performing because they take pride in their ability and they enjoy what they do.

The issue is that the only way someone can know they’re really good at  using Boolean queries for talent discovery and identification is if they have a basis of comparison. Sure – they may get good results, but how do they really know their skill level unless they have a chance to see what others would do if they were working the same positions?

If you were raised on a desert island in complete isolation and you taught yourself how to play the guitar – how would/could you know if you were actually any good at playing the guitar?

The reason why I won’t share any more of my theories as to why the experiment didn’t get more responses is that I’m actually more curious about your thoughts as to why there was such a low response rate. If you have any ideas, please share them.

My Analysis and Search Strategy

I was going to hold out for more responses, but I don’t think I would be accomplishing anything by doing such.

So, without further ado, here is the job description from the experiment:

Business Analyst

  • This mission critical role will involve you working with the inventory team to provide data analysis, reporting and technical expertise to meet business objectives. You will work directly with the inventory control group to provide the technical needs as driven by the business, and you will be required to provide business analysis support to the eCommerce and retail groups.

Required:

  • A minimum of 3 years of experience as a Business Analyst
  • Strong data analysis skills
  • Crystal Reports experience preferred
  • BSCS or related degree and/or experience
  • Experience with enterprise systems

My Analysis/Observations

While I don’t typically prefer to search by title, “Business Analyst” is a fairly common title used to describe this kind of work. However, I am aware that other companies may call people performing this role many things, including “business systems analyst,” “system analyst,” “systems analyst,” “requirements analyst,” “functional analyst,” “IT analyst,” or simply “analyst” or “consultant.”

You may be surprised to learn that my testing on this position specifically, 50% of the results mentioned “business analyst” and none of these (“systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”). The other 50% mentioned one of these (“systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”), and NOT “business analyst.”

The even 50/50 split threw me off, but I checked it 3 times. :-) That means that if you only searched for the title “business analyst,” you would be missing 50% of the available results, and you wouldn’t even know it.

From the job description, I can see this person will be primarily responsible for data analysis and reporting, and likely using Crystal Reports for the reporting, otherwise it wouldn’t likely be listed as preferred experience. Also of note is that this person will be working with the inventory control group as well as supporting the eCommerce and retail groups.

While experience with enterprise systems is listed as a requirement, I would not likely including it in my search efforts without speaking with the hiring manager/team for more clarification. There is no way to know if the reference to “enterprise systems” means something like experience with PeopleSoft, Oracle E-Business, SAP, etc., or something else. Outside of the major ERP systems, “enterprise systems” isn’t something many people will explicitly state in their resume or LinkedIn profile, and searching for it (or simply “enterprise”) can also result in many false positives. I’ll assess this qualification on the phone with the people I find.

(Some of) My Searches

I would choose to search my ATS and Monster first, given that I can take advantage of proximity search, the sheer volume of the sample size (millions to 10’s of millions – and there aren’t as many resumes on the Internet as you might think) and data/information depth, which is a weakness of LinkedIn.

For my first searches, I would not use any date range – I would search all resumes.

Search #1

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and retail and (ecommerce or “e-commerce” or b2b or b2c)

Search Notes:

I thought about searching for the common title denominator of “analyst,” but my experience and some testing has shown me that that results in many false positives (e.g., programmer analyst).

Even though the job description doesn’t require or even desire it, the ideal candidate would have some retail experience. However, this is very tricky to isolate, as most people won’t explicitly mention “retail,” and any retail experience they have will often represented in the companies they have worked for (e.g., Nordstrom’s, which has online retail, aka e-commerce, btw).

You can’t practically search for every retail company in existence, and using a “retail” industry filter on any site will only yield a small fraction of all people who actually do have retail industry experience.

Similar to the retail experience, even though it is not required or even preferred, I chose to search for (ecommerce or “e-commerce” or b2b or b2c), simply because if that’s the kind of group that this person will be responsible for supporting, why not try first to find people who have done it before? While those are some of the most common ways to express e-commerce experience, I could have added EDI, e-tail*, e-business, electronic funds tranfer, online transaction processing, credit card payments, etc., to the “or” statement.

I chose to search directly for the term “inventory” first, with the realization that some people may simply mention supply chain management/SCM as the only hint in their resume/profile that they may have experience with inventory systems. As such, I would plan on taking this approach if necessary after I exhaust all “inventory” searches. However, I also know that just because someone mentions supply chain management, it does not automatically mean they have inventory systems experience – there are plenty of other aspects/stages of SCM than just inventory.

I did not search for “Crystal Reports” because some people will only make reference to “Crystal Enterprise,” Crystal v.X, SAP Crystal, or Crystal Xcelsius. Thus I chose to work with the common denominator of “Crystal” at the expense of a few false positive hits of people with the name Crystal (who will, by the way, have everything else I am searching for). Crystal Reports is fairly common and widely used, however, if I didn’t find enough people with Crystal Reports, I would likely search for other popular reporting applications (“business objects,” “reporting services,” SSRS, Cognos, Hypersion Essbase, etc.)

This search only returns a very small number of results (which is exactly what I wanted – the bulls eye) – so once I have cherry-picked all of the people I want to contact that have obvious clues that they have everything my manager/client could possibly ask for (even though they didn’t explicitly ask for some – but I aim to exceed expectations by always searching for “maximum qualifications” first).

After I’ve gone through the first search’s results, I begin to systematically NOT out the non-required skills and experience, never seeing the same results twice, until my last search is limited to the minimum qualifications.

Search #2

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and not retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #3

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #4

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #5

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and not retail and (ecommerce or “e-commerce” or b2b or b2c)

Search #6

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #7

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and inventory and not retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search #8

(“business analyst” or “systems analyst” or “system analyst” or “data analyst” or “requirements analyst” or “functional analyst”) and crystal and report* and analy* and data near analy* and not inventory and not retail and not (ecommerce or “e-commerce” or b2b or b2c)

Search Progression

After searching sources that allow me to leverage proximity search and have a huge volume of deep data, I would then go to other resume databases (no proximity and the same or smaller volume of deep data), then LinkedIn (large volume of mostly shallow data), then the Internet.

However, I have to say I likely would not even have to leave my own database and go to a job board, LinkedIn, or the Internet because I would have found a decent quantity of very nice matches that I would begin messaging, contacting, assessing, matching and recruiting.

I also would not likely get past search #3 or #4 to find 20-30+ people to call. This is important because every person I would contact would have at least 2 out of the 3 implicitly desired skills (implied by the job description, but not explicitly stated in the requirements).

Final Thoughts

There is no single “best” search string for any given position, although one could argue that some search strategies and approaches are better than others, in terms of more quickly getting to and uncovering people are are highly likely to be a very strong match for a given role, taking required and desired (explicitly mentioned as well as implicit) skills and experience into consideration.

You should always wonder about whether or not your search strategy and specific queries are uncovering the strongest potential candidates that you have access to. The fact that every search “works” is a significant yet oft-overlooked issue, giving many the false perception that whomever is returned from the queries is all that is available to be found.

Perception is reality for most, but subjective reality isn’t objective reality – there are always more people to be found than are actually found by sourcers and recruiters. Always.

I will likely post another Boolean search string experiment in the future – please share your searches and encourage others to do the same.

Thanks!