An increase in new security requirements from both regulatory and cyber insurance sources has led to increased awareness and demand for penetration tests. To meet this demand, many new vendors are offering fully automated penetration tests that are touted as both affordable and effective. To evaluate these assertions, we procured automated pen tests from three vendors and will discuss the cost and effectiveness of these current automated pen test platforms. We will highlight the advantages and limitations of automated pen testing services.