Browse
···
Log in / Register

Java Developer with Web Crawler Experience

Negotiable Salary

Axiom Software Solutions Limited

Austin, TX, USA

Favourites
Share

Description

Role: Java Developer with Web Crawler Experience Location: Austin TX(Hybrid) Responsibilities: 1. Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources. 2. Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.). 3. Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible. 4. Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites. 5. Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations. 6. Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g., robots.txt, copyright laws). Requirements: Proficiency in Java and experience with Java-based web scraping libraries (e.g., Jsoup, Apache HttpClient). Knowledge of web crawling frameworks and tools, such as Scrapy, Selenium, or Puppeteer. Strong understanding of HTML, CSS, JavaScript, and web data structures. Familiarity with data parsing and handling techniques for JSON, XML, and other common formats. Experience with database technologies (SQL, NoSQL) to store and manage scraped data. Knowledge of HTTP protocols, headers, proxies, and load handling.

Source:  workable View original post

Location
Austin, TX, USA
Show map

workable

You may also like

Workable
Senior Backend Platform Engineer, Distributed Systems
Help redefine how the DoD makes multi-billion-dollar force-design decisions. In this role, you'll fuse physics-driven simulation, interactive computing, and verified AI code-generation to create next-generation wargaming platforms. If building ultra-low-latency APIs, taming high-volume geospatial data, and leading with clean, production-ready Python gets you fired up, let’s talk. What you’ll do Own the service layer—design, build, and scale FastAPI micro-services and lightning-fast ZeroMQ messaging pipelines running in Kubernetes and bare-metal clusters. Wrangle data at speed & scale—shape and query multi-TB Postgres/PostGIS datasets, orchestrate Redis for sub-millisecond state, and keep everything rock-solid under bursty load. Glue the stack together—expose crisp, well-versioned REST & WebSocket endpoints for the frontend crew and simulation kernel. Ship continuously—automate CI/CD, observability, and security hardening to DoD standards; push to prod with confidence. Lead by doing—drive code reviews, mentor teammates, and set the standard for test coverage and documentation. Why Code Metal? Mission with impact: your APIs become the nervous system of digital battlefields influencing multi-billion-dollar defense acquisitions. Velocity: tight-knit teams, weekly releases, zero bureaucratic drag. Ownership: no passengers—every engineer ships code that matters. Requirements Must-have credentials 4+ years building production backends in modern Python, with deep FastAPI (or equivalent async framework) experience. Proven expertise in ZeroMQ, NATS, Kafka, or similar high-throughput messaging systems. Hands-on with Postgres/PostGIS and Redis in performance-critical workloads. Cloud-native chops—Docker, Kubernetes, and one major provider (AWS, GCP, Azure, or GovCloud). Active Secret clearance or eligibility to obtain one.  Bonus points C++ or Rust skills for bridging high-performance simulation modules. Hardened services for FedRAMP / STIG compliance. Observability mindset—Prometheus, Grafana, OpenTelemetry. TS/SCI clearance. Benefits Health care plan with 100% premium coverage, including medical, dental, and vision. 401k with 5% matching. Paid Time Off (Uncapped Vacation, plus Sick & Public Holidays). Flexible hybrid work arrangement. Relocation assistance for qualifying employees.
Boston, MA, USA
Negotiable Salary
Workable
Data & BI Senior Data Engineer
Job Description: We are seeking a highly skilled and experienced Senior Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with a specialization in Matillion, SSIS, Azure DevOps, and ETL processes. This role will involve designing, developing, testing, and deploying ETL jobs, collaborating with cross-functional teams, and ensuring efficient data processing. Key Responsibilities: Design, develop, test, and deploy Matillion ETL jobs in accordance with project requirements. Collaborate with the Data and BI team to understand data integration needs and translate them into Matillion ETL solutions. Create and modify Python code/components in Matillion jobs. Identify opportunities for performance optimization and implement enhancements to ensure efficient data processing. Collaborate with cross-functional teams, including database administrators, data engineers, and business analysts, to ensure seamless integration of ETL processes. Create and maintain comprehensive documentation for Matillion ETL jobs, ensuring knowledge transfer within the team. Create, test, and deploy SQL Server Integration Service (SSIS) packages and schedule them via Active Batch scheduling tool. Create Matillion deployment builds using Azure DevOps CI/CD pipeline and perform release manager activities. Review code of other developers (L2, L3-BI/DI) to ensure code standards and provide approval as part of code review activities. Resolve escalation tickets from the L2 team as part of the on-call schedule. Working knowledge of API and Postman tool is an added advantage. Qualifications: Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 5+ years of experience in data engineering, with a focus on ETL processes. Proficiency in Matillion, SSIS, Azure DevOps, and ETL. Strong knowledge of SQL, Python, and data integration techniques. Experience with performance optimization and data processing enhancements. Excellent collaboration and communication skills. Ability to work in a fast-paced, dynamic environment. Preferred Skills: Experience with cloud platforms such as AWS or Azure. Knowledge of data warehousing and data modeling. Familiarity with DevOps practices and CI/CD pipelines. Strong problem-solving skills and attention to detail.
Atlanta, GA, USA
Negotiable Salary
Cookie
Cookie Settings
Our Apps
Download
Download on the
APP Store
Download
Get it on
Google Play
© 2025 Servanan International Pte. Ltd.