0%
LogicNet: A Self-Improving
Neural Logic Framework
‘Using Bittensor to improve Bittensor’
LogicNet’s Core Contributions
Open Source Model Development
Development and maintenance of LogicNet-7b, an open-source specialized 7B parameter model optimized for logical reasoning
Continuous model refinement
Public accessibility via Hugging Face
Dynamic Dataset Evolution (Aristotle - The Red Pajamas of Logic)
Creation of the world’s largest, ever-expanding, curated dataset for logical reasoning
Development of synthetic data generation protocols via dual-task mining
Implementation of validator-driven quality control mechanisms
Dynamic Time-Test Compute System
Development of complexity-aware computation systems
Implementation of dynamic time-test compute windows based on query complexity
Implementation of intelligent time-allocation protocols for optimal response generation
Creation of standardized metrics for computational requirements in logical reasoning tasks
These contributions represent significant advancements in decentralized AI development, particularly in the domain of logical reasoning and efficient computation. Through these innovations, LogicNet will establish new standards for both model performance and resource utilization in the AI ecosystem.
The LogicNet Approach
LogicNet represents a pioneering approach to creating self-improving logical reasoning systems, leveraging Bittensor's decentralized infrastructure to drive continuous improvement. Our innovative system harnesses miner incentives to expand and enhance logical reasoning capabilities, while maintaining a unique balance between efficiency and performance.
Addressing the Challenge
Miners naturally optimize their models to maximize rewards, often leading to overfitting on existing datasets. However, by designing a system that channels miners’ profit-seeking motivation into dataset expansion, we can transform a potential limitation into a mechanism for continuous improvement.
Our Solution
LogicNet addresses this challenge by implementing a three-task system that:
Demonstrates competence on existing queries generated from Aristotle, our ever-growing dataset for logical reasoning
Contributes to Aristotle’s expansion through the creation of new synthetic queries
Utilizes miner tasks to respond to validator queries, create new synthetic queries, and respond to queries generated by other miners
Key Components
Miner Incentives: Miners are incentivized to perform tasks that contribute to the expansion and improvement of Aristotle, driving continuous improvement in logical reasoning capabilities.
Validator-Driven Quality Control: Validators assess and cache the best answers, ensuring data quality and maintaining the integrity of Aristotle.
Self-Improving Ecosystem: Our system creates a self-improving ecosystem where miners’ natural incentives drive the enhancement of the entire network’s capabilities through continuous dataset enrichment and validation.
By combining these key components, LogicNet creates a unique framework that drives continuous improvement in logical reasoning capabilities, while maintaining a compact and efficient model size.
Eliminating Overfitting
LogicNet’s unique approach to logical reasoning eliminates overfitting by continuously expanding and enriching the training dataset, Aristotle. Through our three-task system, miners respond to validator queries generated from Aristotle, generate new synthetic queries, and respond to peer-to-peer miner query generation, creating a dynamic and diverse dataset that reduces the risk of overfitting. By leveraging miner incentives to drive dataset expansion, we ensure that our models are trained on a constantly growing and evolving dataset, preventing them from becoming overly specialized to a single dataset and reducing the likelihood of overfitting.
Validator Contribution
Validators play a crucial role in eliminating overfitting through LogicNet’s implementation of model rotation. Validators randomly select one of the designated models to use when creating an initial query for miners to respond to and then create new synthetic queries from. This random selection process eliminates the miners’ ability to overfit by knowing in advance which model the validator will run. By introducing this element of uncertainty, we prevent miners from tailoring their models to a specific validator model, further reducing the risk of overfitting, while also enriching the Aristotle ever-growing dataset, giving miners multiple high-quality models to create synthetic queries from.
Minimizing Overfitting Risk
The combination of our dynamic dataset expansion, miner incentives, and validator model rotation creates a robust system that minimizes the risk of overfitting. By continuously challenging our models with new and diverse data, and introducing randomness in the validation process, we ensure that our models remain adaptable and resilient to overfitting. This approach enables LogicNet to achieve high performance in fine-tuning, as miners are incentivized to continuously refine and adapt their models to the evolving dataset and validation process.
Solving For Overfitting - Explained
The solution addresses the overfitting problem through a carefully designed feedback loop between miners and validators:
Validator-Driven Quality Control
Dataset Expansion Process
Validators randomly sample questions from the Zebra Bench dataset
Miners generate new questions based on these samples
Validators assess and cache the best answers
High-quality examples are logged into the Weights & Biases storage
Selected examples are curated and pushed into the Aristotle on Huggingface once a week
Starting Pool of Data Sets:
The following data sets will be used by LogicNet, that miners will use to generate the synthetic datasets / queries for other miners to answer:
- Mathgenerator (https://github.com/lukew3/mathgenerator)
- allenai/ZebraLogicBench-private (grid_mode & mc_mode) (https://huggingface.co/datasets/allenai/ZebraLogicBench-private)
- openbmb/UltraInteract_sft (https://huggingface.co/datasets/openbmb/UltraInteract_sft)
- openai/gsm8k (main) (https://huggingface.co/datasets/openai/gsm8k)
- TIGER-Lab/MMLU-STEM (https://huggingface.co/datasets/TIGER-Lab/MMLU-STEM)
- mcaleste/sat_multiple_choice_math_may_23 (https://huggingface.co/datasets/mcaleste/sat_multiple_choice_math_may_23)
Two-fold Incentivisation
Miners incentives will now come from two sources:
- Higher quality responses - encouraging miners to improve their models’ logical and reasoning capabilities.
- Higher quality synthetic query generation.
Anti-Overfitting Mechanism
The system prevents overfitting in two key ways:
Continuous Dataset Growth
Miners must train on both the original Zebra Bench and new synthetic data.
As the synthetic dataset grows, the total training data expands.
Larger, more diverse datasets naturally reduce overfitting potential.
Quality-Driven Selection
Validators maintain quality control through answer caching, keeping only the best answers.
Only the best-performing examples make it into the synthetic dataset.
This ensures dataset growth doesn’t compromise quality.
Dataset management and pruning
Continuous de-duplication of Aristotle to ensure unique queries/responses.
Continuous monitoring and pruning of any potentially bad or low quality data.
Keep all prior versions of Aristotle to allow the ability to rollback in case of any issues.
System Architecture
Core Components
Multiple miners performing specialized tasks.
Set up an ever-growing Aristotle dataset on huggingface.
A foundational pool of datasets.
Dynamic weights and biases storage (WanDB).
Validator model rotation.
Validator grading miner responses to queries and miner synthetic query generation.
Self-Improvement Mechanism
The framework leverages miners' natural incentive to maximize rewards by channeling their optimization efforts into dataset enhancement. Rather than viewing overfitting as a problem, LogicNet transforms this tendency into a tool for dataset expansion.
Similar to RedPajamas' role in LLM training, LogicNet aims to become the definitive dataset for logical reasoning.
Innovation in Logic Research
Dataset Growth Strategy
The framework employs a unique approach where miners must train on both existing and newly generated data, creating a positive feedback loop that:
Reduces overfitting through dataset expansion
Increases statistical diversity
Improves generalization capabilities
Quality Assurance
Validators ensure dataset quality by:
Randomly selecting tasks for evaluation
Maintaining a cache of best answers
Logging and validating new question-answer pairs
Time-Test Compute
LogicNet's Time-Test Compute framework is designed to optimize computation time and resource allocation for logical reasoning tasks. Our approach involves a combination of complex query detection, adaptive time windows, and data weighting systems to ensure efficient and effective computation.
Complex Query Detection
Our system uses pattern recognition to identify complex logical structures in incoming queries, analyzing question depth, dependencies, and logical steps required to determine the optimal computation time. We utilize validator caching to track computation patterns and inform our adaptive time windows.
Adaptive Time Windows
We deploy dynamic time allocation based on question complexity, scaling compute windows proportionally to logical depth. This approach ensures that complex queries receive the necessary computation time to produce accurate results, while simpler queries are processed more quickly.
Data Weighting System
Our data weighting system tags complex questions through validator assessment and stores complexity metadata in Weights & Biases. We apply weighted importance during synthetic dataset curation to ensure that complex queries are given priority in the training process.
Implementation Strategy
Our implementation strategy involves developing complexity scoring metrics for incoming queries, tracking computation time versus accuracy correlations, and building complexity profiles for different question types. We balance computation time with answer quality, implementing early stopping for simpler queries and scaling resources dynamically for complex problems. Our models are fine-tuned to recognize computation requirements, and we develop efficient resource allocation strategies and specialized handling for high-complexity queries.
By optimizing time-test compute, LogicNet is able to efficiently process complex logical reasoning tasks while maintaining high accuracy and performance.
Summary - Impact on AI Research
LogicNet positions Subnet 35 at the forefront of logical reasoning research by:
Creating the largest, high-quality logic dataset, Aristotle, which is continuously expanded and enriched through our dynamic dataset evolution process
Establishing new benchmarks in logical inference through our innovative approach to logical reasoning, which combines complex query detection, adaptive time windows, and data weighting systems to optimize computation time and resource allocation
Contributing to the broader field of AI research by demonstrating the potential of decentralized AI systems to drive continuous improvement in specialized AI capabilities, and by providing a robust framework for logical reasoning that can be applied to a wide range of domains and applications.
This framework represents a significant advancement in decentralized AI systems, demonstrating how properly aligned incentives can drive continuous improvement in specialized AI capabilities, and how a combination of technical innovations and decentralized architecture can lead to breakthroughs in logical reasoning and AI research.